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FOREWORD 

This  is  the  fifth  of  six  volumes  which  record  aspects  of  the  planning,  financing,  design, 
construction,  and  operation  of  the  California  State  Water  Project. 

The  State  Water  Project  conserves  and  distributes  water  to  much  of  California's  population  and 
irrigated  agriculture.  It  also  provides  electric  power  generation,  flood  control,  water  quality 
control,  new  recreational  opportunities,  and  enhancement  of  sports  fisheries  and  wildlife  habitat. 

Construction  of  the  first  phase  of  the  State  Water  Project  was  completed  in  1973.  The  $2.3 
billion  reimbursable  cost  is  being  repaid  by  the  water  users  and  other  beneficiaries.  It  is  expected 
that  another  $0.7  billion  will  be  spent  during  the  next  decade  to  construct  authorized  facilities 
for  full  operation. 

This  volume  discusses  aspects  of  the  planning,  design,  construction,  and  operation  of  the 
various  control  systems  associated  with  the  Project.  Though  the  facilities  of  the  Project  are  varied 
and  located  throughout  the  State  of  California,  they  must  be  operated  as  a  single  system  and  must 
be  coordinated  with  the  operations  of  other  agencies.  This  operation  is  carried  out  through 
systems  which  have  been  integrated  projectwide  to  provide  remote  monitoring  and  control  of 
power  and  pumping  plants,  dams  and  reservoirs,  aqueduct  control  structures,  and  various  other 
hydraulic  features  throughout  the  Project.  The  system  represents  a  sophisticated  application  of 
monitoring  and  control  procedures  on  a  scale  never  before  used  on  water  conservation  and 
transportation  projects. 

The  other  volumes  give  the  details  of  the  history,  design,  construction,  and  operation  of  the 
Project.  The  subjects  are:  Volume  I,  History,  Planning,  and  Early  Progress;  Volume  II,  Convey- 
ance Facilities;  Volume  III,  Storage  Facilities;  Volume  IV,  Power  and  Pumping  Facilities;  and 
Volume  VI,  Project  Supplements. 


/v  a^^^^'xJ-w^ 


John  R.  Teerink,  Director 
Department  of  Water  Resources 
The  Resources  Agency 
State  of  California 


Copies  of  this  bulletin  at  $3.00  each  may  be  ordered  from: 

State  of  California 

DEPARTMENT  OF  WATER   RESOURCES 

P.O.  Box  388 

Sacramento,  Californio  95802 

Moke   checks   payable  to   STATE  OF  CALIFORNIA 
California  residents  add  sales  tax. 
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ABSTRACT 

This  volume  describes  the  planning,  design,  construction,  and  operation 
activities  which  occurred  during  the  period  of  implementing  the  control  system 
for  the  State  Water  Project. 

Early  system  planning  indicated  that  more  efficient  and  reliable  methods  of 
control,  other  than  those  in  existence  elsewhere,  were  necessary  for  the  operation 
of  the  State  Water  Project.  This  planning  also  indicated  that  fully  automated 
control  was  both  economically  and  operationally  justified,  and  should  be 
considered.  A  control  system  plan  was  conceived  to  develop  the  system  and 
implement  it  concurrent  with  the  placing  into  operation  of  the  conveyance  and 
storage  facilities  of  the  Project. 

An  important  feature  of  the  planning  of  a  control  system  to  operate  the  State 
Water  Project  was  the  implementation  of  an  Automation  Pilot  Model  on  the 
South  Bay  Aqueduct.  This  Model  served  as  a  prototype  to  evaluate  various 
control  system  configurations  and  concepts,  evaluate  the  plan  of  operation,  and 
plan  the  optimum  control  system  for  the  State  Water  Project. 

Subsequent  to  design,  but  concurrent  with  construction  of  the  South  Bay 
Aqueduct  Control  System  Model,  an  Operations  Control  Plan  was  developed  to 
define  the  structure  of  the  projectwide  control  system  and  the  operational 
relationships  between  the  individual  facilities  to  be  controlled,  that  is,  detailed 
operating  criteria  for  all  features  of  the  Project.  These  criteria  served  as  the  guide 
for  the  design  of  the  projectwide  control  system. 

The  actual  design  and  construction  of  the  Project  Control  System  took  place 
between  1964  and  1974.  All  significant  portions  of  the  work  to  construct  the 
control  systems  for  the  Project  were  performed  by  contractors  under  the 
provisions  of  the  State  Contract  Act. 

The  communication  systems  required  for  the  operation  of  the  State  Water 
Project  are  comprehensive  and  extensive.  These  communications  have  been 
leased  from  telephone  industries  within  California.  Communication  systems  are 
one  of  the  most  critical  elements  in  the  Project  Control  System. 
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CHAPTER  I.     INTRODUCTION 


Major  decisions  made  early  in  the  planning  of  the 
State  Water  Project  dictated  the  need  for  a  responsive 
and  reliable  control  and  monitoring  system.  The  de- 
sign of  project  features  and  the  proposed  method  of 
operations  ruled  out  a  local  manual  type  of  operation 
used  in  many  other  water  conservation  and  transpor- 
tation projects. 

Though  the  facilities  of  the  Project  are  varied,  and 
located  throughout  the  State  of  California,  they  must 
be  operated  as  an  integrated  system  and  must  be 
coordinated  with  the  operations  of  other  agencies. 
Figure  1  shows  the  location  of  the  various  features  of 
the  Project. 

The  control  system  consists  of  instrumentation, 
communications,  computers,  and  other  electronic 
equipment  to  provide  for  the  remote  monitoring  and 
control  of  17  pumping  and  power  plants  with  119 
hydroelectric  units,  71  check  structures  with  198 
radial  gates,  and  other  miscellaneous  facilities. 

To  facilitate  the  control  of  these  facilities,  the 
Project  is  divided  into  five  separate  control  areas:  (1) 
Oroville-Thermalito,  (2)  Delta,  (3)  San  Luis,  (4)  San 
Joaquin,  and  (5)  Southern  California.  Each  area  has  a 
control  center  from  which  remote  control  of  all 
facilities  within  that  control  area  is  performed  and 
supervised. 

A  Project  Operations  Control  Center,  located  in 
Sacramento,  is  responsible  for  the  management  and 
coordination  of  all  operations  of  the  Project.  This 
center  also  serves  as  a  back-up  control  center  for  any 


of  the  aqueduct  area  control  centers.  This  redundancy 
is  essential  to  provide  the  high  reliability  required  by 
the  project  control  systems. 

Three  basic  categories  of  control  systems  are 
provided  throughout  the  Project.  They  are: 

1.  Aqueduct  Control  Systems — Systems  which 
interconnect  the  Project  Operations  Control  Center 
and  the  area  control  centers  with  remote  sites 
throughout  the  Project. 

2.  Plant  Control  Systems — Local  systems  within 
the  plants  which  provide  for  local  and  remote  control 
of  specific  pumping  and  power  plant  functions 
necessary  for  operation  of  the  plant. 

3.  Miscellaneous  Control  Systems — Systems  which 
provide  for  remote  control  and  monitoring  of  other 
hydraulic  features  throughout  the  Project,  including 
spillways,  reservoirs,  intakes  and  outlets,  and 
turnouts. 

In  this  volume,  the  planning,  design,  construction, 
and  operation  of  the  various  control  systems  are 
discussed. 

This  volume  is  oriented  to  the  noncontrol  system 
engineer  or  interested  public  works  official  who  may 
be  faced  with  a  similar  need  to  implement  such 
control  systems.  To  the  practicing  professional 
control  system  engineer  or  technician,  it  will  serve  as 
an  orientation  guide  to  point  the  direction  of  the 
Department  of  Water  Resources'  efforts  in 
development  of  a  control  system. 


CHAPTER  II.     PLANNING  CONSIDERATIONS 


During  the  formulation  of  the  State  Water  Project, 
an  examination  was  made  of  existing  operational 
methods  used  in  other  similar  projects  throughout  the 
United  States.  It  was  concluded  that  more  efficient 
and  reliable  methods  of  control  than  those  in  existence 
elsewhere  were  necessary  for  the  operation  of  the 
State  Water  Project.  In  addition,  process  control  and 
systems  equipment  had  advanced  considerably  and 
were  being  continually  improved.  This  made  it 
desirable  to  evaluate  the  use  of  advanced  equipment 
and  techniques  to  control  operation  of  State  Water 
Project  facilities. 

In  1961,  the  Department  started  an  investigation  of 
methods  of  controlling  the  operation  of  the  State 
Water  Project  with  particular  emphasis  on  the 
following  items: 

1.  An  investigation  of  total  system  requirements. 

2.  A  review  of  proposals  presented  by  various 
control  systems  firms. 

3.  An  investigation  of  work  in  this  field  done  by 
other  agencies. 

4.  A  review  of  the  Department's  needs,  not  only  for 
control  systems  but  also  for  other  management  and 
operational  needs. 

Operational  Constraints 

Operational  concepts  and  requirements  imposed 
certain  constraints  on  the  method  of  control.  There 
were  three  requirements  which  were  basic  to  the 
proposed  method  of  operating  the  Project.  These 
requirements  were  to:  (1)  meet  contractual 
agreements  of  water  customers  by  use  of  the 
"controlled  volume  concept"  of  operation,  (2) 
provide  for  minimum  on-peak  operation,  and  (3) 
respond  immediately  to  adverse  operating  conditions 
or  emergencies. 

Controlled  Volume  Concept  of  Operation 

The  operational  philosophy  which  was  developed 
to  meet  water  contractual  obligations  has  been 
designated  the  "controlled  volume  operation".  This 
concept  of  operation  permits  a  rapid  change  of  flow  in 
the  canal  reaches  of  the  California  Aqueduct  to  be 
accomplished  with  a  minimum  of  water  surface 
fluctuation.  This  concept  was  dictated  by  special  and 
unique  physical  characteristics  of  the  Project. 

Controlled  volume  operation  is  a  concept  which 
utilizes  the  philosophy  of  nearly  simultaneously 
operated,  projectwide,  aqueduct  facilities.  The  water 
level  in  the  Aqueduct  is  checked  to  full  design  depth 
at  all  times.  When  a  change  in  flow  is  required, 
pumping  units  are  started  or  stopped  and  check  gates 
raised  or  lowered  simultaneously.  As  additional  pump 
units  are  started  or  stopped,  check  gates  again  are 
raised  or  lowered  simultaneously.  By  operating  the 
facilities  in  this  manner,  flow  in  the  Aqueduct 
increases  or  decreases  without  appreciably  changing 
the  volume  of  water  in  the  canal  cross  section.  This 


approaches    pipeline    flow    conditions    in    the   open 
aqueduct. 

The  main  stem  of  the  California  Aqueduct  is  444 
miles  in  length,  of  which  391  miles  are  canal,  12  miles 
are  tunnel,  and  41  miles  are  pressure  pipeline. 
Consequently,  water  customer  service  requires  special 
consideration,  since  it  takes  a  particle  of  water 
approximately  15  days  to  travel  that  distance, 
depending  on  the  velocity  of  flo\V  in  the  Aqueduct. 
Controlled  volume  aqueduct  operation  provides  the 
capability  to  make  a  maximum  delivery  change 
rapidly  through  the  Aqueduct. 

San  Luis  Reservoir,  O'Neill  Forebay,  and 
Silverwood  Lake  provide  the  major  facilities  for 
regulating  the  water  en  route  through  the  main 
aqueduct.  Controlled  volume  operation  minimizes  the 
need  for  offstream  storage,  spill  basins,  or  wasteways. 

Limitations  of  water  surface  fluctuations  add  to  the 
difficulty  of  moving  water  through  the  Aqueduct. 
The  design  of  the  Aqueduct  has  placed  certain 
restrictions  on  its  operation  with  regard  to  maximum 
allowable  water-level  fluctuations.  In  the  canal 
portions  of  the  Aqueduct,  the  lining  consists  of 
4-inch-thick  unreinforced  concrete  panels  without 
underdrains.  To  reduce  the  risk  of  lining  failure,  flow 
changes  in  the  Aqueduct  must  be  controlled  normally 
within  the  following  criteria: 

1.  Water  surface  fluctuations  must  not  exceed  6 
inches  in  one  hour  or  1  foot  in  any  24-hour  period. 

2.  If  a  6-inch  drop  is  taken  in  any  one  hour,  the 
remaining  6  inches  must  be  distributed  uniformly 
over  the  other  23  hours. 

3.  The  initial  6-inch  fluctuation  may  be  taken  in 
less  than  one  hour. 

Controlled  volume  operation  reduces  the  adverse 
effects  of  hydraulic  transient  conditions  in  the 
Aqueduct.  Water  surface  fluctuations  are  reduced  to 
almost  half  that  of  the  conventional  method  of 
aqueduct  operation. 

Over  half  of  the  water  delivered  through  the 
Aqueduct  is  for  municipal  and  industrial  use,  which 
requires  operation  throughout  the  year  with  delivery 
of  high-quality  water  and  a  minimum  of  outages.  A 
highly  efficient  year-round  delivery  capability  with 
minimum  manpower  costs  is  necessary. 

Certain  cross-drainage  structures  located  along  the 
Aqueduct  were  designed  to  divert  water  into  the  canal 
during  periods  of  high  runoff.  This  inflow  could  cause 
adverse  operating  conditions.  The  concept  of  control 
had  to  provide  capability  to  react  immediately  to 
correct  this  condition,  thereby  keeping  damage  to  a 
minimum  and  restoring  the  operation  to  normal  in 
minimum  time. 

Minimum  On-Peak  Operation 

Pumping   costs    during   on-peak    power   demand 


hours  are  considerably  higher  than  similar  pumping 
costs  during  off-peak  hours.  For  this  reason,  minimum 
pumping  during  on-peak  hours  was  adopted  as  an 
operational  objective. 

On-peak  hours  are  the  hours  of  the  day  in  which 
demand  for  power  by  nonproject  users  is  the  greatest. 
These  hours  are  from  0700  to  2200  hours,  Monday 
through  Friday,  and  from  1300  to  2200  hours  on 
Saturdays. 

Pumping  operation  outside  these  hours  provides 
the  required  pumping  at  the  least  cost.  This  pumping 
scheme  places  great  reliance  on  the  method  of  control, 
since  flow  modifications  are  greater  in  magnitude  and 
occur  more  often  within  this  compressed  time  period. 

Response  to  Emergencies 

Several  natural  physical  characteristics  affect  the 
Aqueduct  and  make  it  susceptible  to  damage  and 
failure. 

The  Aqueduct  crosses  a  number  of  faults  and, 
throughout  most  of  its  length,  is  in  a  seismically  active 
area.  Failure  or  rupture  of  some  portions  of  the 
Aqueduct  at  some  time  is  possible.  The  method  of 
control  must  have  the  capability  of  responding  rapidly 
to  isolate  the  appropriate  reach  by  stopping  the  flow 
downstream  from  the  failure  and  reducing  the  flow 
upstream  to  an  amount  equal  to  the  water  customer 
deliveries  upstream  from  the  failure. 

Through  the  San  Joaquin  Valley  and  Antelope 
Valley,  the  Aqueduct  traverses  areas  that  are  bisected 
by  few  natural  drainage  channels  of  sufficient 
capacity  to  carry  full  flow  of  the  Aqueduct.  For 
wasteways  to  effectively  handle  emergency  flow  from 
the  Aqueduct,  it  would  have  been  necessary  to 
construct  expensive  detention  basins  or  excessively 
long  channels  to  take  the  water  to  the  San  Joaquin 
River  or  similar  drainage  courses.  An  alternative,  and 
more  economical  approach,  was  to  include  check 
structures  at  sufficient  spacing  to  contain  the  water  in 
the  Aqueduct  during  these  emergencies.  The  use  of 
check  structures  to  create  basins  in  the  Aqueduct,  in 
combination  with  controlled  volume  operation, 
provides  the  capability  to  perform  the  function  of 
controlling  excess  water  in  the  Aqueduct.  This  is 
accomplished  by  rapidly  reacting  to  any  adverse 
operating  conditions  through  the  aqueduct  control 
facilities  and  immediately  adjusting  the  check 
structures  and  pumping  plants. 

Other  Control  Requirements 

In  addition  to  operational  constraints,  it  was 
determined  that  the  control  philo.sophy  for  the  Project 
had  to  satisfy  other  control  requirements.  These  were: 

1.  High  degree  of  reliability. 

2.  Integrated  operation  of  the  facilities  of  the  State 
Water  Project. 

3.  Coordinated  operation  with  associated  water  and 
power  projects. 

4.  Centralized  dispatching  of  water  and  power. 


5.  Provision  for  systematic  and  flexible  growth. 

Reliability 

The  method  of  control  adopted  for  application  to 
the  Project  had  to  be  inherently  reliable.  Economic 
loss  would  result  should  a  malfunction  disturb  the 
operating  regimen  of  the  power  generation  features. 
In  additon,  damage  and  possible  loss  of  life  could 
occur  due  to  malfunction  of  control  features  in  the 
open-channel  portions  of  the  California  Aqueduct, 
particularly  through  the  highly  developed  San 
Joaquin  Valley  and  the  densely  populated  areas  in 
Southern  California.  Thus,  the  method  of  control  of 
the  system  had  to  provide  a  high  degree  of  reliability. 

Integrated  Operation  of  Facilities 

The  following  guidelines  were  established  for 
integrated  operation  of  the  facilities.  The  State  Water 
Project  would  consist  of  varied  features  whose 
interdependence  would  be  such  that  completely 
integrated  operation  was  a  requisite. 

Oroville-Thermalito  complex  would  be  operated 
for  long-term  carryover  storage  as  well  as  power  gen- 
eration. Releases  from  Oroville  would  be  controlled 
primarily  to  meet  water  demand  schedules,  power 
generation,  and  water  quality  control  in  the  Sacra- 
mento-San Joaquin  Delta.  Three  pump-generation 
units  at  Thermalito  and  Edward  Hyatt  Powerplants 
would  provide  for  off-peak  pumpback  capability  to 
maximize  on-peak  power  generation. 

San  Luis  Reservoir  would  provide  seasonal  storage 
capacity  for,  and  would  regulate  flows  in,  the 
California  Aqueduct.  Pumping  into  the  Reservoir  and 
releases  from  it  would  be  integrated  with  operations 
to  the  north  and  south. 

The  California  Aqueduct  would  be  operated  for 
both  water  transportation  and  conservation.  The 
Aqueduct,  from  San  Luis  Reservoir  to  the  Tehachapi 
Mountains,  would  present  a  difficult  problem  in 
control.  Since  it  would  be  very  expensive  to  provide 
spillways,  spill  basins,  or  off-canal  storage,  and  since 
canal  storage  would  be  limited  by  freeboard  and 
allowable  water  surface  fluctuation,  close  and 
simultaneous  control  over  several  reaches  of  the 
Aqueduct  would  be  a  necessity. 

A  succession  of  pumping  plants  on  the  north  side  of 
the  Tehachapis  and  power  recovery  facilities  on  the 
southern  slopes  would  require  that  reactions  to  a 
departure  from  steady  state  operation  in  this  area  be 
rapid  and  general  in  nature.  The  aqueduct  operation 
as  far  north  as  San  Luis  could  be  affected. 

The  complex  interrelationship  between  all  project 
features  would  preclude  any  extended  operation  of 
one  independently  of  the  others. 

Coordinated  Operation  with  Associated  Projects 

Coordinated  operation  with  other  water  and  power 
agencies    would    be    a    requirement    for    efficient 


operation  and  best  utilization  of  resources.  Further 
guidelines  were  established  for  this  type  of  operation. 

One  requirement  would  be  the  scheduling  and 
coordination  of  power  generated  and  consumed  by 
the  Project  with  participating  power  utilities.  The 
upstream  reservoirs  of  the  State  Water  Project  and  the 
Federal  Central  X^alley  Project  would  be  operated, 
insofar  as  possible,  in  a  coordinated  manner.  This 
coordinated  operation  would  provide  economic  and 
operational  benefits  by  increasing  the  firm  power 
yield  of  the  combined  generation  capabilities  through 
optimization  of  water  releases  to  satisfy  water 
demands  of  service  areas  and  power  demands  at  load 
centers. 

Further  coordination  would  be  required  between 
the  State  Water  Project  and  the  Federal  Central 
V^alley  Project  through  operation  of  the  San  Luis 
Joint-Use  Facilities.  The  Department  would  operate 
these  joint-use  facilities  to  serve  water  to  both  federal 
and  state  service  areas  upon  a  predetermined  water 
delivery  schedule. 

Centralized  Dispatching 

In  accordance  with  the  basic  requirements  previ- 
ously stated,  a  centralized  point  of  monitoring  and 
dispatching  of  overall  project  operation  would  be 
necessary.  This  project  control  central  would  serve  to 
coordinate  the  operation  of  all  of  the  facilities  of  the 
entire  Project.  Scheduling  and  dispatching  informa- 
tion would  be  transmitted  to  the  control  central  from 
the  operating  facilities  of  the  integrated  systems. 
These  data  would  be  analyzed  and  reduced  to  provide 
optimum  control  and  operation  of  the  facilities. 

The  control  central  would  receive  requested  water 
delivery  schedules  for  state  service  areas  as  well  as  a 
schedule  from  the  U.  S.  Bureau  of  Reclamation  for 
deliveries  to  federal  service  areas. 

In  addition,  operational  data  concerning  upstream 
storage  and  power  generation  capabilities  would  be 
transmitted  into  this  control  central.  Participating 
power  utilities  would  receive  power  schedules  for 
project  pumping  and  transmit  schedules  for  project 
power  generation  to  and  from  this  facility. 

Provisions  for  Expansion 

The  growth  and  development  of  California  and  the 
long  operating  life  of  the  State  Water  Project  made  it 
imperative  that  provision  for  systematic  and  orderly 
expansion  of  control  be  a  consideration  in  adoption  of 
any  control  method.  As  operating  experience  was 
gained  and  changes  in  operational  concepts  became 
necessary,  the  method  and  equipment  employed  for 
control  would  have  to  be  flexible  and  adaptive. 

With  these  guidelines  in  mind,  the  Department 
embarked  on  an  investigation  of  various  control 
concepts  which  could  be  employed  for  the  State  Water 
Project. 


Control  Concepts 

Five  control  concepts  were  investigated  within  the 
operational  constraints  and  control  requirements. 
These  five  concepts  were: 

Manual  Control 

Conventional  Control 

Remote  Control 

Semiautomated  Control 

Fully  Automated  Control 

To  compare  these  concepts  on  a  common  basis,  each 
was  defined  and  its  application  to  the  Project  for  both 
normal  and  emergency  operation  was  described. 

It  was  assumed  that  the  system  or  process  pertained 
only  to  State  Water  Project  facilities.  All  functions 
which  would  be  conducted  at  departmental  headquar- 
ters, such  as  contractor  billing  and  accounting  for  wa- 
ter and  power,  were  excluded  from  these  descriptions. 
In  all  cases,  it  was  assumed  that  adequate  mobile  radio 
communication  capabilities  existed. 

Manual  Control 

Manual  control  was  defined  as  direct  on-site  control 
and  supervision  of  all  features  in  the  system. 
Operators  would  make  the  physical  control 
manipulations  at  the  sites  required  to  achieve  the 
operational  objectives. 

Operators  would  be  required  to  read  staff  gauges, 
manipulate  the  release  mechanisms,  and  measure  and 
read  the  released  flow  rates  at  the  reservoirs 
throughout  the  system.  As  the  operator  performed 
these  duties,  he  would  log  his  actions. 

The  operator  assigned  to  generating  plants  would 
be  required  to  operate  and  monitor  each  unit  from  the 
unit  control  boards.  In  the  case  of  pumping  plants, 
relatively  the  same  conditions  would  apply. 

In  the  open-channel  portions  of  the  Project,  water 
operators  (ditch  riders)  would  be  required  to  regulate 
aqueduct  checks  and  maintain  correct  flow  rates 
through  service  area  turnouts.  The  operators  would 
be  responsible  for  operating  up  to  40  miles  of  canal 
involving  such  duties  as  raising  and  lowering  the 
check  gates  to  maintain  a  near-constant  water  surface, 
reading  and  logging  the  values  on  staff  gauges,  and 
measuring  and  setting  the  flow  rate  in  the  canal.  In 
addition,  they  would  be  responsible  for  setting 
turnouts  to  deliver  the  required  flow  at  each  turnout 
in  his  area  of  responsibility.  They  would  be 
responsible  for  maintaining  and  servicing  the  on-site 
rcording  devices,  the  records  from  which  would  be 
sent  to  department  headquarters  for  billing 
computations. 

Manual  control  would  require  a  large  number  of 
operators  throughout  the  system.  These  operators,  in 
order  to  achieve  a  degree  of  efficient  operation,  would 
require  considerable  training.  However,  it  was 
foreseeable  that  a  number  of  these  trained  operators 
would  be  required  for  norpial  operation  only. 
Consequently,  should  an  emergency  condition  arise, 
some    control     reactions     would     be    directed     by 


untrained  or  unfamiliar  personnel  who  normally 
would  have  been  assigned  other  duties.  It  was 
apparent  that  the  factor  of  each  individual  operator's 
unique  reaction  to  a  system  emergency  could  cause 
unreliable  operation. 

The  application  of  the  concept  of  manual  operation 
to  emergency  situations  would  require  that  a  written 
set  of  instructions  be  supplied  to  all  operators  for  use 
in  emergencies.  These  instructions  would  attempt  to 
eliminate  operational  errors  in  the  system. 

Should  an  emergency  condition  arise  in  the  canal, 
the  situation  could  conceivably  go  unnoticed  for  a 
long  period  of  time.  Once  the  emergency  became 
evident,  the  reaction  time  or  the  time  required  for  the 
water  operators  to  effect  control  actions  could  be 
lengthy. 

Conventional  Control 

Conventional  control  was  defined  as  that  method 
which  utilized  intervening  supervisory  equipment 
between  the  operator  and  the  device  to  be  controlled. 
For  example,  within  a  project  pumping  or  generating 
plant,  conventional  supervisory  equipment  would 
allow  the  plant  operator  to  automatically  start  or  stop 
units  from  a  control  room  within  the  plant.  The 
supervisory  equipment  would  sequentially  check 
permissive  points,  start  the  auxiliaries,  and  start  the 
unit.  In  addition  to  start-up  and  shutdown  operation, 
the  supervisory  equipment  would  also  monitor 
conditions  during  operation.  These  monitored 
quantities  would  be  displayed  on  a  control  panel 
located  in  the  control  room  of  the  plant. 

Conventional  control  of  the  checks  and  turnouts  in 
the  open-channel  portions  of  the  Project  would  utilize 
controllers  to  provide  local  automatic  control.  The 
operator  would  insert  a  water  level  setpoint  into  this 
controlling  device,  which  would  then  control  the 
structure  about  this  setpoint  for  an  indefinite  period. 

Reservoir  water  surface  elevation  readings  and  gate 
controls  would  be  reported  electronically  to  a 
reservoir  control  center  commonly  included  in  the 
generating  plant  control  room.  These  readings  would 
be  made  by  floats  or  other  water  surface  measuring 
devices  equipped  with  telemetering  units.  Gate 
controls  would  be  arranged  to  allow  for  remote 
operation. 

Conventional  control  would  be  coordinated 
through  area  control  centers.  These  area  control 
centers  would  have  radio  contact  with  water  operators 
in  the  field,  as  well  as  radio  or  telephone  contact  viith 
pumping  plant  operators.  At  these  area  control 
centers,  system  dispatching  and  data  logging  would  be 
accomplished  by  manual  means  after  reception  of  the 
daily  master  schedule  from  the  control  central  located 
at  department  headquarters. 

In  emergency  situations,  the  conventional  supervi- 
sory equipment  would  function  to  protect  tKe  pump- 
ing and/or  generating  units  from  malfunction. 
However,  should  the  supervisory  equipment  itself 
malfunction,  the  plant  operator  would  take  complete 
and  direct  manual  control  of  each  unit.  Since  these 


failures  should  occur  infrequently,  the  system  would 
be  operating  with,  at  best,  unfamiliar  if  not  untrained 
personnel  in  emergency  situations. 

In  the  open-channel  portions  of  the  system,  since  no 
indications  would  be  telemetered  to  a  common 
location,  should  an  emergency  condition  arise,  the 
time  lapse  between  the  occurrence  of  an  emergency 
and  its  detection  could  be  great.  Once  the  emergency 
condition  became  apparent,  the  required  travel  time 
of  the  water  operator  to  correct  the  situation  would 
extend  this  period. 

Should  a  power  failure  occur  at  a  pump  turnout,  the 
resultant  rejected  flow  would  tend  to  raise  the  water 
surface  at  the  checks.  As  the  \\  ater  surface  rose,  the 
local  automatic  control  device  at  the  check  would 
increase  the  check  opening  to  maintain  a  constant 
water  surface  upstream  from  the  check.  As  the  check 
opened,  the  flow  through  it  would  increase.  Rejected 
flows  would  proceed  downstream  until  the  capacity  of  j_ 
the  canal  was  no  longer  able  to  contain  them,  at  \Ahich 
time  they  would  overflow  the  canal. 

The  same  condition  would  exist  at  a  gravity 
turnout;  that  is,  if  there  were  to  be  a  malfunction,  the 
device  itself  would  be  unable  to  indicate  such  a 
malfunction  to  the  water  operator  or  apply  temporary 
corrective  measures.  Therefore,  it  was  apparent  that 
warnings  of  turnout  malfunctions  might  come  from 
sources  other  than  the  project  operation  staff. 

Remote  Control 

Remote  control  was  defined  as  that  method  which 
utilized  conventional  supervisory  equipment  in 
pumping  or  generating  plants,  and  terminal 
equipment  and  local  automatic  control  devices  in  the 
open-channel  portions  of  the  Aqueduct  which  could 
be  controlled  from  a  remote  location.  That  is, 
pumping  and  generating  plants  would  be  controlled 
through  the  use  of  conventional  supervisory 
equipment  which  would  provide  for  control  from  a 
remote  common  operation  center.  The  facilities  on 
the  canal  would  be  controlled  by  an  operator  located 
at  the  same  control  center  using  either  radio  or 
land-line  communications  to  transmit  and  receive 
control  instruction  and  information. 

As  discussed  in  the  section  on  conventional  control, 
upstream  reservoirs  would  be  controlled  remotely 
from  the  generating  plant  control  room.  Water 
surface  indication,  gate  positions,  and  quantities  of 
flow  would  be  displayed  in  the  control  room  for  use 
by  the  operator.  The  generating  plant  would  be 
controlled  by  operators  using  conventional 
supervisory  equipment. 

In  the  case  of  pumping  plants,  some  plants  would  be 
controlled  remotely  from  other  plants  using  conven- 
tional supervisory  equipment  with  indications  and 
control  instructions  being  received  and  transmitted 
via  either  radio  or  land-line  communications. 

Remote  control  of  the  canal  would  require 
equipment  at  checks  and  turnouts  which  would 
respond     to     remotely     transmitted     signals.     This 


equipment  would  have  the  capability  of  recognizing  a 
coded  message  containing  an  instruction,  converting 
that  instruction  into  an  action  by  the  control  device, 
and  retransmitting  back  to  the  initiating  location 
confirmation  that  the  coded  message  was  received  and 
that  action  had  been  taken. 

With  remote  control,  the  ability  to  react  to 
emergency  situations  would  be  greatly  enhanced  over 
that  of  the  methods  previously  discussed.  Indications 
of  generating  or  pumping  unit  status  would  be 
immediately  available  at  ail  times  in  the  control  room. 
Should  the  indicators  show  that  an  emergency 
situation  was  impending,  the  particular  troubled  unit 
could  be  shut  down  and  a  maintenance  man 
dispatched  before  the  actual  trouble  occurred. 

In  the  canal,  it  would  be  possible  to  foresee  an 
impending  emergency  condition  through  the  use  of 
remote  indication.  In  an  emergency,  such  as  a  power 
outage  at  a  pump  turnout,  the  condition  of  this  pump 
would  be  remotely  indicated  at  the  control  center  and 
would  therefore  provide  instantaneous  notification  of 
its  outage  to  the  water  operator.  The  operator  would 
then  be  able  to  cope  with  the  rejected  turnout  flow  by 
the  regulation  of  checks  up-  and  downstream  from  the 
outage  area. 

In  case  of  a  failure  of  on-site  or  terminal  equipment, 
which  would  receive  and  transmit  messages  to  and 
from  the  control  center,  the  operator  would  receive  an 
instantaneous  indication.  A  technician  would  still 
have  to  be  dispatched,  however,  to  repair  the 
equipment  item  that  failed,  thus  resulting  in  a 
comparable  travel  time  to  that  mentioned  in  the 
previous  discussions. 

Semiautomated  Control 

Semiautomated  control  of  the  State  Water  Project 
would  utilize  the  capabilities  of  a  process  control 
computer  or  processor  to  effect  control  of  upstream 
storage  reservoirs,  generating  plants,  and  pumping 
plants  in  the  system  and  to  receive  data  from  on-site 
sensors  in  the  open-channel  portions  of  the  California 
Aqueduct  to  provide  guidance  for  the  operator  in  the 
control  center. 

In  the  operation  of  upstream  reservoirs  and  generat- 
ing plants,  the  control  processor  having  responsibility 
for  that  particular  facility  would  accomplish  the  re- 
quired scheduling  within  a  master  schedule  as  trans- 
mitted from  a  control  center  and  would  release  the 
required  flows  from  the  storage  reservoirs  for  genera- 
tion. The  control  processor  would  also  function 
through  conventional  supervisory  equipment  to 
schedule  and  operate  the  pumping  plants  located 
throughout  the  system. 

Open-channel  portions  of  the  California  Aqueduct 
would  be  controlled  by  an  operator  receiving 
guidance  from  a  programmed  control  processor.  The 
processor  would  receive  data  from  the  sensing  devices 
at  the  control  structures  in  the  aqueduct  system  and 


would  make  these  data  available  for  use  by  the 
operator.  All  ultimate  control  responsibility  would  be 
placed  in  the  hands  of  the  operator,  however. 

The  concept  of  semiautomated  control  would 
provide  for  emergency  conditions  in  the  system.  The 
processor  would  receive  indications  from  the 
conventional  supervisory  equipment  located  at 
generating  and  pumping  plants.  These  indications 
would  provide  for  an  analysis  by  the  processor  and 
would  be  followed  by  a  diagnosis  of  the  impending 
emergency  condition.  By  utilization  of  the  processor's 
capability  to  schedule  the  pumping  units,  not  only  for 
operation  but  maintenance  purposes  as  well,  it  was 
conceivable  that  emergency  conditions  would  not 
occur  as  frequently  as  under  the  previously  discussed 
concepts  of  control. 

Fully  Automated  Control 

Fully  automated  control  would  involve  the  intro- 
duction of  closed-loop,  real-time  control  computers  or 
processors  as  controlling  devices  into  all  control  loops 
required  for  optimum  operation  of  the  State  Water 
Project.  The  extent  to  which  operators  would  enter 
into  actual  control  of  these  facilities  would  be  limited 
even  in  extreme  emergency  cases.  The  control  system 
structure  was  assumed  to  be  a  large  control  central 
located  in  Sacramento  supervising  the  functions  of 
five  area  control  centers  located  in  Oroville,  near  the 
Delta  Pumping  Plant,  at  the  San  Luis  complex,  near 
Bakersfield,  and  in  Southern  California. 

These  area  control  centers  would  supervise  the 
operation  of  that  portion  of  the  system  under  their 
jurisdiction.  An  operator  would  be  on  duty  at  these 
area  control  centers  at  all  times  as  would  operators 
(water  and  power  dispatchers)  at  the  control  central 
in  Sacramento. 

Scheduling  and  optimizing  of  pumping  and  gener- 
ating plants  would  be  directly  accomplished  by  com- 
puters or  processors.  These  processors  would  function 
within  parameters  and  master  schedules  transmitted 
to  them  by  the  control  central  in  Sacramento.  Opera- 
tion of  pumping  and  generating  plants  would  be 
through  the  use  of  conventional  supervisory  equip- 
ment, with  monitoring  of  supervisory  indicators  done 
by  the  processors.  The  actual  hourly  generation  quan- 
tities would  be  those  defined  by  the  participating  pub- 
lic utilities  in  the  dispatching  schedule  coordinated 
through  the  control  central.  Power  consumption 
would  be  handled  in  a  like  manner  for  the  system 
pumping  plants. 

In  the  open-channel  portions  of  the  system,  during 
normal  operation,  radial  gate  checks  would  utilize 
local  automatic  control  about  limits  or  setpoints 
within  schedules  as  determined  by  the  area 
processors.  These  schedules  would  be  determined 
from  the  master  schedule  transmitted  to  the  area 
control  center  from  control  central.  The  area  control 
center  processor,  however,  would  have  an  override 
option  enabling  it  to  assume  control  should  the  local 
loop  device  malfunction. 


Real-time  data  from  the  system  would  be  examined, 
processed,  and  analyzed;  any  system  deviations  noted; 
commands  issued  to  correct  those  deviations  and 
prevent  future  occurrences;  and  data  logged  for 
storage  to  constitute  essential  project  records. 

Analysis  and  Comparison  of  Costs 

Each  of  the  five  control  concepts  were  rated  as  to 
expected  levels  of  performance  in  specific  areas.  Table 
1  shows  the  relative  standings  with  a  maximum 
weight  of  5  and  a  minimum  of  1.  The  column  totals 
indicate  the  overall  level  of  performance  which  could 
be  expected  of  each  operational  concept  relative  to  the 
others. 


Table   1.      Relative  Expected  Levels  of  Performance  of 
Various  Control  Concepts 


PERFORMANCE 

CONTROL  CONCEPT 

AREA 

Man- 
ual 

Con- 
ven- 
tional 

Re- 
mote 

Automated 

Semi 

Fully 

Reliability . 

1 
1 
1 

1 

1 
1 

1 

2 
2 
2 

2 
2 
2 
2 

4 
4 

3 

4 

3 
4 

5 

5 
4 
4 

4 
4 
4 

5 

5 

5 

Optimum  Operation 

Extent  of  Immediate  Con- 
trol.. 

5 
5 

5 

Customer  Service 

Warning  of  Emergency. . . 

5 
S 

TOTAL    . 

7 

14 

27 

30 

35 

Savings  were  estimated  on  the  basis  of  the  number 
of  operating  positions  which  would  not  have  to  be 
filled  throughout  the  system.  These  savings  consisted 
only  of  salaries  and  wages  and  did  not  include  the 
costs  of  recruitment  and  training  of  personnel  for 
these  positions.  The  costs  included  a  phasing  concept 
which  allowed  for  sufficient  staffing  at  each  area 
control  center  to  allow  portions  of  the  Project  to 
become  operational  in  an  orderly  manner  as 
construction  was  completed.  Once  operational,  it  was 
assumed  that  some  personnel  remained  with  that 
portion  while  the  bulk  of  the  staff  moved  to  a 
succeeding  portion.  This  phasing  operation  was 
assumed  to  last  for  a  period  of  three  years  for  each 
completed  portion. 

This  comparison  showed  that  the  salary  savings 
derived  would  be  adequate  to  repay  the  costs  of 
installing  automation  equipment.  In  1962,  when  this 
estimate  was  made,  the  present  worth  of  the  direct 
incremental  capital  costs  for  automation  equipment  at 
4%  interest  was  estimated  to  be  $7,094,191.  The 
present  worth  of  costs  for  maintenance  and  replace- 
ment of  this  equipment,  extended  over  a  .'>0-year  pay- 
out period  at  4%  interest,  totaled  $6,388,126. 

Savings  derived  from  decreased  personnel 
requirements  for  fully  automated  control,  compared 
to  the  conventional  control  computed  on  the  same 
basis,  totaled  $26,380,943. 

The  benefit/cost  ratio  was: 


Total  Savings 
Total  Costs 
$26,380,943 
$13,482,317 


$26,380,943 


5,388,126  +  $7,094,191 


=  1.957 


Factors  included  in  deriving  the  aforementioned 
costs  were  as  follows: 


An  economic  comparison  was  also  made.  However, 
since  a  comparison  of  each  individual  control  concept 
would  present  difficulties  in  the  accuracy  of 
assessment  and  assignment  of  costs,  a  comparison  was 
made  based  only  on  two  concepts,  conventional  and 
fully  automated  control  (Table  2). 

The  costs  for  conventional  control  were  used  as 
base  costs,  that  is,  costs  for  terminal  and  other  equip- 
ment which  would  be  used  for  fully  automated  con- 
trol were  considered  to  be  incremental.  The  personnel 
requirements  for  conventional  control  also  were  uti- 
lized as  base  requirements.  The  estimated  equipment 
costs  were  based  on  proposals  submitted  to  the  De- 
partment by  control  systems  firms.  Costs  were  includ- 
ed for  communication  lines  from  the  Sacramento 
control  central  to  each  area  control  center  and  from 
each  area  control  center  to  the  features  requiring  con- 
trol. I  his  communication  and  data  gathering  system 
was  composed  of  both  land-line  and  microwave  links. 
The  costs  as  shown  did  not  reflect  any  use  of  the 
existing  State  of  California  integrated  microwave  sys- 
tem but,  instead,  were  based  on  a  completely  separate 
system. 


Costs  of  Conventional  Control 

It  was  assumed  that  the  total  system  was  staffed 
with  personnel  to  accomplish  required  activities  in 
the  following  categories: 
General  Maintenance  Unit 

Plant  Maintenance 

Canal  Maintenance 

Ground  and  Reservoir  Maintenance 
Operations  Unit 

Plant  Operation 

Canal  Operation 
Administrative  Unit 

Costs  for  personnel  in  the  above  categories  included 
the  following: 

Personnel  Salaries 

Equipment 

Supplies,  Materiels,  and  Tools 

Administrative  and  General  Expenses 

Costs  of  Fully  Automated  Control 

Equipment  costs  for  a  fully  automated  system  were 
derived   from  several   itemized   quotations  received 


from  equipment  suppliers  and  included  substantial 
allowances  for  installation,  programming,  and  sys- 
tems engineering,  as  well  as  system  hardware. 

It  was  assumed  that  yearly  maintenance  costs  would 
approximate  5%  of  the  initial  equipment  costs.  In 
addition,  an  equipment  replacement  allowance  was 
included  plus  an  allowance  for  software.  This 
replacement  allowance  was  included  in  the  form  of  a 
sinking  fund  of  25-year  life  at  4%  interest.  Table  2 
summarizes  the  comparison  of  cost  between  fully 
automated  and  conventional  operation  of  the  Project. 


Conclusions  and  Recommendations  on  Control 
Concepts 

It  was  concluded  from  the  analysis  that  fully 
automated  control  of  the  State  Water  Project  was  both 
economically  and  operationally  justified. 

Another  conclusion  was  that  this  ultimate  control 
concept  be  phased  as  the  features  of  the  State  Water 
Project  were  constructed.  The  decision  was  made  that 
as  each  facility  was  constructed,  that  facility  would 
initially  be  operated  by  either  conventional  or  by 
remote  methods.  As  experience  was  gained  with 
remote  control,  groups  of  facilities  would  be  phased 
gradually  into  semiautomated,  and  then  into  fully 
automated,  control. 

The  following  program  was  adopted  in  1962  for  the 
design  and  implementation  of  a  control  system  for  the 
State  Water  Project.  This  program  was  divided  into 
four  phases: 

Phase  1.  Automation  Pilot  Model 

Phase  2.  Preliminary       System       Design       and 

Performance  Specifications 
Phase  3.  Final  System  Design  and  Procurement 

Specifications 
Phase  4.  Equipment    Procurement    and    System 
Implementation 

Phase  1.  Automation  Pilot  Model 

An  important  feature  of  the  entire  analysis  made  to 


select  a  control  system  to  operate  the  State  Water 
Project  was  the  implementation  of  an  Automation 
Pilot  Model  on  the  South  Bay  Aqueduct.  This  Model 
would  serve  as  a  tool  to  evaluate  various  control 
system  configurations  and  concepts,  evaluate  the  plan 
of  operation,  and  plan  the  optimum  control  system  for 
the  State  Water  Project.  The  Model  would  function 
under  actual  operating  conditions  on  a  real-time  basis, 
illustrating  the  concepts  of  centralized  control. 

A  consulting  engineering  firm  would  be  retained  to 
assist  with  the  development,  testing,  and  evaluation  of 
the  Model. 

The  consultants  would  determine  what  organiza- 
tional requirements  within  the  Department  would  be 
necessary  to  successfully  engage  in  the  various  phases 
of  automation.  Also,  maximum  and  minimum  partici- 
pation by  the  Department  would  be  set  forth. 

Phase  2.  Preliminary  System  Design  and  Perform- 
ance Specifications 

A  systems  engineering  firm  qualified  in  the  fields  of 
civil  engineering,  computers,  and  control  systems,  but 
without  any  product  line,  would  be  selected  for  the 
purpose  of  assisting  the  Department  with  the  work 
required  under  Phase  2  of  the  program.  The 
consultant  would  supply  the  necessary  engineering 
services,  materials,  resources,  and  facilities  to  conduct 
engineering  investigations  and  to  design  and  establish 
the  preliminary  system  for  the  control  system  of  the 
State  Water  Project.  This  would  be  done  in 
accordance  with  operational  criteria  furnished  by  the 
Department. 

In  addition,  performance  specifications  would  be 
prepared  and  furnished  for  each  major  item  of  equip- 
ment required  in  the  total  system  design.  The  per- 
formance specifications  were  to  express  requirements 
in  the  form  of  input,  output,  function,  and  operational 
characteristics  of  items  of  equipment  required  to  sat- 
isfy the  system  design,  leaving  the  details  of  design, 
fabrication,  and  internal  functions  of  the  equipment 
to  the  discretion  of  the  Department. 

The  data  processing  and  control  system  covered  by 
these  specifications  would  include  all  data  handling 


Table  2.     Comparison  of  Costs  Between  Fully  Automated  and  Conventional  Control 


Area  Control  Center 

Present  Worth  of 

Annual  Savings 

(1) 

Present  Worth  of 

Cost  for  Additional 

Equipment* 

(2) 

Present  Worth  of 
Costs  for  Maintenance 

and  Replacement  of 

Additional  Equipment 

(3) 

Total  Present  Worth 

of  Additional  Costs 

(2)  +  (3) 

(4) 

?3,641,687 
4,411,868 
2,739,019 
3,033,255 
8,387,214 
4,167,900 

8707,037 
1,364,198 
1,012,152 
1,127,709 
1,794,683 
1,088,412 

5640,914 
1,231,397 

921,782 
1,023,798 
1,594,212 

976,023 

51,347,951 

2,595,595 

Delta 

1,933,934 

2,151,507 

3,388,895 

Southern  California 

2,064,435 

Total 

?26,380,943 

57,094,191 

56,388,126 

513,482,317 

•  Costs  of  equipment  over  and  above  that  required  for  conventional  control. 


equipment  from  the  source  of  output  of  the  various 
turnouts  and/or  distribution  points,  the  monitoring 
of  certain  equipment  functions,  data  assessment  and 
evaluation,  recording,  computing,  and  reporting  and 
display  to  operation  personnel  for  both  real-time  and 
deferred-time  modes  of  operation. 

Part  of  the  design  considerations  would  include 
communication  requirements  for  data  transmission 
and  control,  in  accordance  with  system  design 
requirements. 

A  complete  set  of  functional  specifications  and  an 
operational  schematic  of  the  total  control  system 
would  be  furnished  under  this  phase  of  the  work. 


of  the  State  Water  Project.  Each  implementation  step 
would  increase  the  overall  capability  of  the  system.  In 
no  case  would  an  implementation  step  decrease  the 
capability  of  the  system.  The  implementation  sched- 
ule would  not  be  tied  to  calendar  dates  but  would 
establish  sequential  steps  of  implementation  by  con- 
sideration of  such  factors  as  cost  estimates  for  each 
step,  importance  and  priority  of  various  portions  of 
the  automated  system,  optimum  efficiency  of  im- 
plementation, and  least  interference  with  the  daily 
operation  of  the  State  Water  Project.  Most  desirable 
grouping  of  implementation  steps  and  optimum  time 
breaks  would  be  indicated. 


Phase  3.  Final    System    Design    and    Procurement 
Specifications 

The  Phase  2  consultant  would  be  retained  to  assist 
the  Department  in  developing  the  final  system  design 
and  procurement  specifications  for  the  control  system 
of  the  State  Water  Project,  if  the  consultant's  perform- 
ance had  been  satisfactory  under  Phase  2  of  the 
proposed  program. 

The  Phase  3  consultant  would  assist  in  performing 
engineering  services  and  supplying  materials,  re- 
sources, and  facilities  necessary  to  develop  the  final 
system  design.  It  was  expected  that  work  performed 
under  Phase  3  might  suggest  or  impose  additional  de- 
sign changes  or  modifications  of  the  Phase  2  design 
effort.  Such  changes  or  modifications  to  the  prelimi- 
nary design  developed  under  Phase  2,  as  deemed 
necessary  and  as  approved  by  the  Department,  would 
be  performed  as  part  of  the  work  under  Phase  3. 

With  the  assistance  of  the  consultant,  procurement 
specifications,  design  drawings,  and  acceptance  test 
procedures  adequate  for  the  Department  to  initiate 
procurement  of  the  total  control  system  for  the  State 
Water  Project  would  be  prepared. 

In  addition  to  developing  procurement  specifica- 
tions of  all  equipment  within  the  control  system,  cost 
estimates  would  be  prepared  for  the  equipment  re- 
quired. 

Implementation  schedules  would  be  formulated 
and  would  consider  all  equipment,  systems,  and  facili- 
ties necessary  to  implement  the  overall  control  system 


Phase  4.  Equipment     Procurement     and     System 
Implementation 

This  phase  of  the  program  would  be  accomplished 
in  accordance  with  the  implementation  plan 
developed  under  Phase  3  and  would  be  accomplished 
under  a  number  of  procurement  contracts.  The 
number  and  scope  of  the  contracts  would  be 
dependent  upon  the  various  systems  or  subsystems  to 
be  installed  at  a  given  time  in  accordance  with  the 
implementation  plan. 

It  was  conceivable  that  one  contract  might  be 
awarded  to  install  one  portion  of  the  system  located  at 
the  northern  section  of  the  State  Water  Project,  two 
or  three  contracts  covering  the  southern  portion  of  the 
Project,  and  miscellaneous  contracts  for  other 
portions  of  the  system.  Where  desirable,  one  contract 
would  be  awarded  to  furnish  complete  major  systems 
encompassing  a  number  of  subsystems.  However,  due 
to  the  long  time  period  involved  in  the  final 
implementation  of  the  Project,  it  was  not  considered 
desirable  that  one  contract  be  awarded  for  the  total 
system. 

The  Department  and  the  consultant  would  monitor 
the  manufacturer  to  assure  that  specifications 
developed  under  Phase  3  were  adhered  to. 

As  discussed  in  subsequent  chapters,  the  four-phase 
program  for  the  design  and  implementation  of  the 
control  system  for  the  State  Water  Project  was 
modified;  only  Phase  1 —  Automation  Pilot  Model — 
was  followed  essentially  as  proposed. 
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CHAPTER  III.     SOUTH  BAY  AQUEDUCT 
CONTROL  SYSTEM  MODEL 


Phase  1  of  the  program  to  develop  the  design  and 
implementation  of  the  control  system  for  the  State 
Water  Project  was  the  Automation  Pilot  Model. 
Initially,  the  "Model"  was  to  have  been  installed  only 
on  the  facilities  of  the  South  Bay  Aqueduct,  extending 
from  South  Bay  Pumping  Plant  through  Patterson 
Reservoir.  This  was  later  expanded  to  include  all 
facilities  of  the  South  Bay  Aqueduct. 

The  information  gained  from  the  test  and 
evaluation  of  the  Model  was  to  have  been  used  by  the 
Department  to  proceed  with  design  and  construction 
of  the  remainder  of  the  project  control  system  in 
accordance  with  Phases  2,  3,  and  4  previously 
described. 

Objectives  of  the  Model 

The  broad  objectives  of  the  South  Bay  Aqueduct 
Control  System  Model  were  twofold: 

1.  Provide  an  adequate  means  to  remotely  control 
and  monitor  the  South  Bay  Aqueduct  from  a  control 
center;  and 

2.  Provide  experience  in  the  design,  procurement, 
installation,  and  operation  of  a  system  having  the 
capability  to  be  used  on  the  entire  aqueduct  system. 

Since  the  requirements  for  remote  control  of  a  small 
aqueduct,  such  as  South  Bay,  were  minor  compared  to 
the  requirements  for  a  large  aqueduct,  it  was  evident 
that  a  system  fulfilling  the  second  objective  would 
more  than  fulfill  the  first. 

Thus,  the  major  design  effort  was  applied  to  the 
objective  of  reproducing  on  a  smaller  scale  a  control 
system,  the  concept  of  which  could  ultimately  be 
expanded  to  control  the  entire  California  Aqueduct. 

To  be  a  suitable  model  and  to  produce  the  informa- 
tion desired,  certain  representative  subordinate  objec- 
tives were  established.  Among  these  u  ere: 

1.  The  number  of  controlled  sites  and  number  of 
functions  at  each  site  would  have  to  be  large  enough 
to  make  the  results  applicable  to  a  more  extensive 
system. 

2.  The  variety  of  functions  and  types  of  controlled 
sites  would  have  to  be  representative  of  future 
requirements. 

3.  The  system  would  have  to  be  computer 
compatible  to  enable  the  testing  of  various  concepts  of 
"operator  guide"  operation  using  a  computer  in  a 
control  center  in  Sacramento. 

4.  Communications  would  have  to  be  designed  for 
high  reliability  and  efficient  use,  and  experience 
gained  would  likewise  have  to  be  transferable  to  the 
larger  system. 

An  additional  objective  of  the  system  was  to  provide 
benefits  to  the  Department  in  the  way  of  enhancing 
its  capability  to  design,  construct,  operate,  and 
maintain  a  sophisticated  control  system.  A  detailed  list 


of  these   anticipated   benefits   was   prepared    before 
construction  of  the  Control  System  Model  was  begun. 

Design  Objectives 

1.  Evaluate  specifications  as  to  conformance  and 
adequacy. 

2.  Make  model  specifications  and  other  pertinent 
data  available  to  designers. 

3.  Enhance  experience  of  control  system  designers. 

4.  Establish  design  criteria. 

5.  Evaluate  equipment  under  environmental  condi- 
tions. 

Construction  Objectives 

1.  Enhance  quality  control  capability. 

2.  Enhance  experience  of  resident  engineers  in 
installation  techniques. 

3.  Improve  acceptance  testing  and  check  out 
experience. 

Operations  and  Maintenance  Objectives 

1.  Evaluate  control  techniques  for  operation. 

2.  Simulate  operational  control  system  problems  or 
malfunctions. 

3.  Determine  optimum  application  of  computer 
control  in  operation. 

4.  Provide  trained  experienced  personnel  for 
California  Aqueduct  operations  activation. 

5.  Provide  operator  and  maintenance  training 
programs. 

6.  Determine  human  factors — man-machine  inter- 
face relationships. 

7.  Evaluate  maintenance  concepts. 

8.  Develop  emergency  control  procedures. 

9.  Develop  safety  procedures. 

10.  Develop  test  control  procedures  for  normal  op- 
eration, component  failure,  equipment  maintenance, 
or  other  critical  situations. 

1 1.  Evaluate  use  of  computer  as  an  off-line  device. 

12.  Establish  water  and  power  dispatching 
procedures. 

13.  Establish  data  logging  and  customer  billing 
procedures. 

14.  Determine  the  number  and  classifications  of 
personnel  required  in  area  control  centers  and 
Sacramento  control  center. 

Consultant  Participation 

One  of  the  first  steps  in  the  Control  System  Model 
Program  was  to  retain  a  consulting  firm  to  assist  the 
Department  in  planning,  designing,  constructing, 
testing,  and  evaluating  the  Model.  In  1962,  the 
Department  required  additional  technical  expertise  in 
the  area  of  computerized  control  systems,  and  it  was 
believed  that  use  of  a  consultant  to  enhance  this 
expertise  was  imperative. 
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At  about  the  same  time,  digital  control  systems 
using  transistor  electronics  had  been  developed  for 
military  and  space  systems  but  had  not  yet  been 
widely  applied  in  industry.  The  Department's 
engineering  personnel  involved  in  the  development  of 
the  proposed  control  system  generally  came  from  an 
industrial  background. 

These  factors  made  it  advantageous  to  seek  the 
advice  of  a  consultant  in  determining  the  best  choice 
of  methods  to  supply  the  required  monitoring  and 
control  capabilities. 

Selection  of  a  consulting  firm  was  made  on  the  basis 
of  experience  in  control  systems  similar  or  applicable 
to  the  one  proposed.  Other  factors  considered  were 
the  willingness  to  work  directly  with  department 
engineers  rather  than  independently  and  the 
understanding  that  the  firm  would  not  be  involved  as 
contractor  in  any  phase  of  system  implementation. 
The  firm  of  Daniel,  Mann,  Johnson,  and  Mendenhall 
of  Los  Angeles,  California,  was  engaged  in  January 
1963. 

Scope  of  the  South  Bay  Model 

The  South  Bay  Aqueduct  branches  from  the  Cali- 
fornia Aqueduct  approximately  2  miles  downstream 
from  the  Delta  Pumping  Plant.  At  this  point,  the 
South  Bay  Pumping  Plant  lifts  the  diverted  water  545 
feet  to  the  top  of  an  adjacent  hill  from  which  the 
Aqueduct  extends  42  miles  through  hill  and  valley 
terrain  to  a  terminus  near  San  Jose.  This  Aqueduct 
with  300  second-foot  capacity  consists  of  32  miles  of 
pipeline,  8  miles  of  open  concrete-lined  canal,  and  2 
miles  of  tunnel.  Figure  2  schematically  shows  the 
facilities  of  the  South  Bay  Aqueduct. 

The  South  Bay  Aqueduct  Control  System  Model,  as 
installed,  provided  complete  control  and  supervisory 
capability  from  two  locations  of  17  separate  facilities 
of  the  South  Bay  Aqueduct,  i.e.,  1  pumping  plant,  7 
check  structures,  3  turnouts,  2  water  quantity 
measuring  meters,  1  blowoff  valve,  1  terminal  storage 
tank,  and  2  reservoirs.  These  facilities  can  be 
controlled  from  either  of  two  control  centers,  one 
located  at  the  Delta  Pumping  Plant  and  the  other 
located  in  Sacramento. 

The  Control  System  Model  was  originally 
developed  for  installation  on  the  South  Bay  Aqueduct 
between  South  Bay  Pumping  Plant  and  Patterson 
Reservoir.  This  concept  included  analog-to-digital 
conversion  at  the  South  Bay  Pumping  Plant,  digital 
data  transmission  between  there  and  Sacramento,  and 
a  control  center  at  Sacramento  with  a  computer 
installed  at  that  facility. 

In  April  1963,  a  decision  was  made  to  extend  the 
Model  through  the  Vallecitos  Turnout  of  the  South 
Bay  Aqueduct  in  order  to  provide  a  reach  of  aqueduct 
with  five  check  structures  in  series  to  test  operational 
concepts.  The  Model  was  extended  to  the  Santa  Clara 
Terminal  in  March  1963  to  ease  the  operational 
problems  and  to  facilitate  water  deliveries. 


The  implementation  plan  called  for  development  of 
the  Model  in  four  stages  as  follows: 

Stage  1 — South  Bay  Aqueduct  System  including 
the  South  Bay  Control  Center. 

Stage  2 — Sacramento  Control  Center. 

Stage  3 — Communication  facilities  between  the 
South  Bay  and  Sacramento  Control 
Centers. 

Stage  4 — Real-Time  Computer  System  at  Sacra- 
mento. 

The  early  stages  of  design  activity  were  devoted  to 
development  of  criteria  and  functional  requirements 
of  the  control  system.  These  criteria  and  requirements 
are  summarized  in  Reference  1  (See  Appendix  A  for 
references) . 

An  important  decision  was  made  in  early  1965  to 
make  use  of  leased  telephone  facilities  for  communica- 
tion rather  than  to  build  a  department-owned  and 
operated  facility.  The  reasons  for  this  decision  and  the 
details  of  the  system  leased  from  the  telephone  com- 
pany are  discussed  in  Chapter  VII,  "Communication 
Systems." 

South  Bay  Aqueduct  Control  System  (Stage  1) 

Design  of  the  South  Bay  Aqueduct  Control  System 
The    equipment    required    for    control    of    the 
Aqueduct  from  the  South  Bay  Control  Center  was 
divided  into  the  following  general  subsystems: 
Site  Equipment  Subsystem 
Communications  Equipment  Subsystem 
Control  Center  Subsystem 
There    were    also    other    items    of    construction 
incidental  to  the  control  system.  Figure  3  shows  a 
functional  block  diagram  of  the  South  Bay  Aqueduct 
Control  System. 

Site  Equipment  Subsystem.  At  each  of  the  17 
sites,  the  conventional  switches  to  turn  motors  on  and 
off,  the  transducers  to  measure  motion  and  position, 
and  automatic  setpoints  and  alarms  to  set  and  measure 
limits  of  movement  were  specified  for  installation  on 
the  equipment  to  be  controlled  or  to  be  monitored. 
These  devices  are  connected  by  cables  to  the  control 
equipment  which  is  located  in  a  small  adjacent 
building.  The  equipment  rack  in  this  building 
contains: 

1.  A  digital  logic  package  to  decode  and  validate 
incoming  messages  and  to  encode  outgoing  messages, 
to  set  up  proper  switching,  to  maintain  alarm 
surveillance,  and  to  convert  any  analog  signals  into 
digital  form. 

2.  Electronic  comparators  and  setpoint  units  to 
provide  automatic  control. 

3.  Transmitters  and  receivers  for  the  communica- 
tion phase. 

4.  A  local  control  panel  to  allow  personnel  visiting 
the  site  to  have  local-manual  control. 
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Gray-code  encoders  are  widely  used  on  the  system. 
Their  ability  to  directly  encode  angular  position  made 
them  ideal  for  gate-position  and  water-level 
measurements.  They  were  also  adopted  as  a  means  of 
encoding  the  rotary  motion  of  pulse-duration 
transmitters  which  had  already  been  installed  for 
pressure  and  water  flow  rate  telemetering. 

Design  called  for  use  of  both  brush  and  optical 
encoders  so  that  both  could  be  tried.  Brush-type 
encoders  were  specified  where  attached  to  the 
telemetering  instruments  because  of  their  low  torque 
requirement  and  high  resolution  for  small  angular 
travel.  The  optical  encoders  were  generally  less 
satisfactory,  principally  because  of  lamp  failures. 

Gate-position  measurement  is  accomplished  by  use 
of  a  tape  and  take-up  reel  driving  a  digital  encoder. 
This  method  is  capable  of  measuring  gate  position  to 
an  accuracy  of  +  0.05  foot  for  gates  having  a  range  of 
travel  of  up  to  6  feet.  On  the  larger  gates  of  the  main 
aqueduct,  which  have  30  feet  of  travel,  this  method 
does  not  have  sufficient  accuracy  even  when  the 
accuracy  requirement  is  reduced  to  +  0.1  foot.  As  a 
result,  a  different  method  of  measuring  gate  position 
was  devised  for  use  on  the  main  aqueduct  (see 
Chapter  \',  "Control  System  Design"). 

Local,  automatic,  water-level  controllers  were  also 
specified  for  the  check  structures.  These  controllers 
compare  the  actual  water  level  at  the  check  with  a 
desired  water  level  setpoint  and,  when  placed  in 
control  of  the  gate,  operate  the  gate  in  the  direction 
required  to  bring  the  water  level  to  the  setpoint.  The 
location  of  the  controlling  setpoint  is  selectable 
between  positions  upstream  and  downstream  of  the 
check  gate.  The  comparator  also  provides  an  alarm 
when  the  level  deviates  from  the  setpoint  by  an 
unacceptable  amount. 

The  controllers  have  two  different  principles  of 
control.  One  type  uses  a  percentage  timer  principle  in 
which  the  gate  is  run  in  the  correcting  direction  for 
a  selected  percent  of  the  time  that  the  water  level  error 
existed  (a  "bang-bang"  controller).  The  other  type 
uses  a  "proportional-plus-reset"  principle  in  which 
the  amount  of  gate  movement  is  determined  by  adding 
together  one  correction  proportional  to  the  amount  of 
error  in  water  level  and  a  second  correction 
proportional  to  the  time  (interval)  of  the  water  level 
error. 

These  local  controllers  were  tested  in  the  Model  to 
determine  if  they  would  be  useful  as  standby  controls 
when  failures  made  it  impossible  to  exercise  remote 
control  or  monitoring. 

Site  equipment  was  initially  installed  in  prefabricat- 
ed sheet  steel  buildings.  These  buildings  proved  un- 
satisfactory because  the  environment  within  them 
caused  the  equipment  to  fail  prematurely.  It  was  even- 
tually necessary  to  insulate  the  buildings  and  add  air 
conditioning  in  order  to  keep  the  equipment  at  a  suita- 
ble operating  temperature.  Figure  4  shows  a  typical 
site  layout. 


Communications  Equipment  Subsystem.  Design 
specified  the  digital  word  used  for  communication 
between  the  control  centers  and  the  sites  and  the 
coding  of  information  contained  within  the  word. 
The  data  rate  is  selectable  at  300,  600,  and  1,000  bits 
per  second.  Although  1,000  bits  per  second  (bps)  was 
desired,  there  was  some  concern  that  communication 
errors  might  be  excessive  at  such  a  high  rate.  Testing 
of  the  various  data  rates  indicated  that  the  1,000  bps 
rate  was  entirely  satisfactory,  and  the  lower  rates  have 
never  been  used.  There  are  four  separate  types  of 
communication  words:  command,  interrogation, 
data-status,  and  alarm.  Each  of  these  is  sent  on  an 
independent  communication  channel.  Commands, 
interrogations,  and  alarms  are  18-bit  words  with  only 
6  bits  of  "function  identification",  while  the 
data-status  word  has  30  bits  and  includes  a  1 3-bit  space 


Figure  4.     Typical    Site    Layout — South    Boy    Aqueduct 
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for  numerical  values.  A  single  parity  bit  and  5-bit 
"Start  of  Message"  code  is  used  for  security.  Gate 
positions,  water  levels,  and  other  quantized  data  are 
received  at  the  control  center,  but  setpoints  cannot  be 
transmitted  to  the  site.  This  requires  continuous 
interrogation  from  the  control  center  when  changing 
the  position  of  gates  or  valves  in  order  to  know  when 
the  desired  position  is  reached  and  the  control  action 
stopped. 

Alarms  are  transmitted  spontaneously  when  the 
event  occurs.  This  requires  a  rather  elaborate 
communication  line-checking  procedure  to  avoid 
placing  more  than  one  alarm  word  on  the  line 
simultaneously. 

Control  Center  Subsystem.  The  control  console 
at  the  South  Bay  Control  Center  provides  the  operator 
with  the  capability  to  control  and  monitor  the  17  sites 
individuallv  by  means  of  four  control  panels  and  to 
monitor  alarms  on  an  alarm  panel.  Pumping  plants, 
checks,  and  turnouts  are  each  controlled  from  an 
individual  panel  while  the  remaining  facilities  are 
grouped  and  controlled  from  the  fourth  panel.  Alarms 
from  all  facilities  are  annunciated  on  an  alarm  panel. 
Figure  5  shows  the  control  center  equipment. 

By  selecting  a  particular  site  to  be  shown  on  the 
appropriate  panel,  the  operator  can  change  operating 
conditions  at  that  site  and/or  obtain  data  or  status 


from  that  site.  The  console  also  contains  an  automatic 
sequencer  that  allows  the  operator  to  set  up 
simultaneous  and  automatic  changes  such  as  gate 
position  or  valve  position  through  setpoint  control.  In 
other  words,  the  operator  can  set  up  an  arrangement 
whereby  a  number  of  gates  or  valves  are  automatically 
positioned  to  a  preset  value. 

A  digital  displav  panel  at  the  control  center  displays 
pertinent  data  items  to  allow  the  operator  to 
continuously  ascertain  current  conditions.  This  panel 
is  automatically  updated  about  once  a  minute. 

1  here  also  is  an  analog  meter  panel  uhich  displays 
some  of  the  data,  such  as  gate  positions  and  water 
levels.  These  same  data  are  also  displayed  on  the 
digital  display  panel.  The  analog  and  digital  displays 
are  shown  in  Figure  6. 

Two,  2-pen  chart  recorders,  which  may  be  selected 
to  record  any  of  the  analog  data,  are  provided.  The 
strip-chart  recorders  are  activated  by  manual  switch- 
ing. 

At  the  heart  of  the  South  Bay  Control  Center  is  a 
small  computer  system.  When  the  contract  for  this 
system  was  being  awarded,  the  "mini-computer"  was 
just  beginning  to  change  the  industrial  control  system 
picture.  Before  that  time,  the  switching  and  logic 
functions  required  for  supervisory  control,  data 
logging,  and  data  display  were  obtained  by  elaborate 


Figure  5.     South   Bay  Control   Center 
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Figure  6.     Analog    ond    Digital    Display    Panel — South    Bay    Control    Center 


wired-logic  systems.  With  the  availability  of 
programmable  devices  at  competitive  cost,  the  use  of 
mini-computers  became  attractive. 

Although  the  South  Bay  Model  was  specified  as  a 
"hard-wire"  system,  that  is,  not  using  a  computer,  a 
qualification  was  added  to  the  specification  permit- 
ting the  use  of  a  "processor"  for  the  control  consoles 
and  display  subsystem  provided  the  contractor  fur- 
nished all  programming  and  other  software  necessary 
to  produce  a  system  functionally  equivalent  to  that 
specified. 

The  computer  system  provides  the  interface 
between  the  control  console  and  displays  and  the 
communications  system.  Figure  7  shows  the  rack 
containing  the  computer,  the  communication 
equipment,  and  the  analog  chart  recorders. 

Quality  Control.  An  important  feature  of  the 
specification  written  for  construction  of  the  control 
system  was  the  inclusion  of  a  number  of  requirements 
to  assure  the  quality  of  the  installation.  These 
requirements  were: 

1.  Prequalification  of  bidders 

2.  Inherent  availability  requirement 

3.  Acceptance  testing 

4.  Operational  availability  demonstration 


The  prequalification  procedure  was  specified  to  as- 
certain whether  a  prospective  bidder  was  technically 
qualified  to  manufacture,  install,  and  test  the  opera- 
tion of  the  equipment.  The  procedure  included  an 
inquiry  into  a  prospective  bidder's  experience,  techni- 
cal personnel,  quality  control,  and  plant  facilities. 
This  prequalification  questionnaire  is  included  in  Ref- 
erence 1. 

The  inherent  availability  requirement  directed  the 
successful  bidder  to  perform  an  availability  analysis  of 
the  system  he  proposed  to  provide.  The  quality  of 
equipment  and  the  amount  of  redundancy  were  re- 
quired to  be  suitable  to  provide  a  96..S%  calculated 
availability.  This  requirement  appears  to  have  served 
its  purpose  by  leading  to  a  judicious  selection  of  relia- 
ble equipment.  At  the  same  time,  it  was  not  so  strin- 
gent as  to  cause  needless  expense  to  the  contractor. 

The  acceptance  testing  requirements  were  not 
unusual.  They  consisted  of  the  normal  checkout  of  all 
the  components  furnished  and  system  operational 
testing. 

The  operational  availability  demonstration  require- 
ment, however,  was  quite  different  from  that  usually 
specified.  It  was  included  to  provide  some  measure  of 


Figure  7.     Computer,    Con 
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control  over  the  contractor  during  construction  to  en- 
sure that  the  Department  received  a  reliable  system. 
The  availability  demonstration  required  the 
Department  to  use  the  system  during  a  six-month 
period  after  it  had  been  installed  and  checked.  During 
this  period  the  system  had  to  be  operational  at  least 
95%  of  the  time  with  a  MTBF  (mean  time  between 
failures)  of  greater  than  50  hours.  The  contractor  was 
to  perform  the  system  maintenance,  with  failures 
being  logged  by  the  Department.  When  failures  were 
too  frequent  or  existed  too  long,  the  contractor  had  to 
continue  the  demonstration  for  a  longer  time. 

Construction  of  the  South  Bay  Aqueduct  Control 
System 

Work  performed  under  Specification  No.  65-3.?, 
South  Bay  Aqueduct  Control  System,  consisted  of 
furnishing  and  installing  a  computer-based  digital 
control  system  on  the  South  Bay  Aqueduct.  This 
system  enabled  the  Aqueduct  to  be  monitored  and 
controlled  from  the  South  Bay  Control  Center  located 
at  the  Delta  Operations  and  Maintenance  Center.  It 
also  provided  for  future  remote  control  -  from  a 
Sacramento  control  center. 

The  work  included  miscellaneous  structural  and 
instrumentation  revisions  and  additions  required  to 


make  the  instrumentation  and  control  equipment 
provided  compatible  with  the  existing  aqueduct 
facilities. 

The  work  also  included  the  calibration  and  testing 
of  all  devices  furnished  and  controlled,  furnishing  of 
system  maintenance  manuals,  maintenance  of  the 
system  for  two  years,  and  training  of  department 
personnel  in  the  operation  and  maintenance  of  the 
system.  Table  3  summarizes  the  contract  data. 

Sacramento  Control  Center  (Stage  2) 

Design  of  the  Sacramento  Control  Center 

An  important  part  of  the  South  Bay  Model  study 
was  the  Sacramento  Control  Center.  This  equipment 
served  as  a  separate  remote  control  center  and  as  the 

Table   3.     Summary  of  the  South  Boy  Aqueduct 
Control  System  Contract  Data 


Specification 

Low  Bid  Amount 

Final  Contract  Cost 

Total  Cost — Change  Orders 

Starting  Date 

Completion  Date 

Prime  Contractor. 


65-33 

S772,460 

551,094,199 

?291,4U 

1/5/66 

12/31/68 

Aetron — Division  of  Aerojet 

General  Corporation 
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interface  between  the  South  Bay  system  and  the  Stage 
4  control  computer,  which  was  to  test  real-time 
computer  control. 

Design  of  this  facility  was  begun  before  the  contract 
for  the  South  Bay  Aqueduct  control  system  was 
awarded  and  was  completed  less  than  a  year  after 
construction  of  the  South  Bay  Aqueduct  system  had 
started.  During  the  early  design  period,  a  minimum 
system  interdependence  was  planned  and  was 
restricted  to  the  communication  line.  The  only  feature 
of  the  interface  which  depended  upon  a  decision  by 
the  South  Bay  Aqueduct  system  contractor  was  the 
selection  of  a  modem  (see  Chapter  \',  Section  on 
Standard  Communication  System).  Other  features  of 
the  interface  had  been  determined  previously  and 
were  thus  included  in  the  design  of  both  facilities. 

The  Sacramento  Control  Center  consisted  of 
consoles  and  displays  for  remote  operation  of  the 
South  Bay  Aqueduct  facilities.  The  consoles  and 
displays  were  located  in  the  Department's  Sacramento 
headquarters  building  adjacent  to  the  Stage  4  control 
computer.  The  Sacramento  Control  Center  was 
specified  to  incorporate  a  computer  as  a  result  of 
AETRON  (the  low  bidder)  electing  to  exercise  the 
computer  option  of  the  Stage  1  specification. 


The  control  console  had  a  separate  console  for  each 
kind  of  controlled  structure;  for  example,  there  was  a 
console  for  pumping  plants,  another  for  check 
structures,  and  so  forth.  The  console  for  check 
structures  had  all  the  controls,  alarms,  and  displays 
necessary  for  check  structures;  and  windows  for 
display  of  gate  positions  with  the  largest  number  of 
gates,  upstream  and  downstream  water  level,  setpoint, 
and  others. 

This  concept  had  some  serious  drawbacks.  One  was 
that  a  console  would  require  provisions  for  any  rare 
readout  or  control,  even  if  it  were  used  at  only  one 
location.  Another  was  the  wide  variety  of  structures 
which  required  a  large  number  of  consoles  and  caused 
operations  to  be  confusing  and  difficult. 

Within  the  limits  of  its  design,  the  Sacramento 
Control  Center  worked  well  with  the  South  Bay 
Aqueduct  Control  System  and  enabled  the  completion 
of  the  testing  and  evaluation  for  which  the  Model  was 
developed.  Figure  8  shows  the  Sacramento  Control 
Center  consoles  and  displays. 

Construction  of  the  Sacramento  Control  Center 

Work  performed  under  Specification  No.  66-26, 
Sacramento  Control  Center,  consisted  of  furnishing 
and  installing  a  computer-based  control  center  corn- 


Figure  8.     Consoles    and    Displays — Sacramento   Control    Center 
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plete  with  operator  console,  display  panels,  cabling, 
and  communication  interfacing  electronics.  This 
equipment  enabled  the  operator  to  control  the  South 
Bay  Aqueduct  from  Sacramento.  The  equipment  was 
installed  on  the  16th  and  17th  floors  of  the  Resources 
Building  in  Sacramento. 

Following  installation,  work  included  testing  to 
demonstrate  operational  capabilities,  maintenance  of 
the  system  for  one  year,  system  maintenance  manuals, 
and  training  of  department  personnel  in  the 
maintenance  and  operation  of  the  system.  Table  4 
summarizes  this  contract. 


Table  4.     Summary  of  the  Sacramento 
Control  Center  Contract  Data 


Specification 

Low  Bid  Amount 

Final  Contract  Cost 

Total  Cost — Change  Orders 

Starting  Date.. 

Completion  Date 

Prime  Contractor 


66-33 
?349,700 
5451,277 
S83,836 
7/11/66 
6/18/70 
Philco-Ford  Corporation 


Communications  Lease  (Stage  3) 

The  communication  circuits  between  the  Sacra- 
mento Control  Center  and  the  South  Bay  Aqueduct 
were  leased  from  the  Pacific  Telephone  and  Tele- 
graph Company.  Details  of  this  lease  arrangement  are 
discussed  in  Chapter  VII,  Communication  Systems. 


Computer  System  (Stage  4) 

Computer  Hardware  Acquisition 

As  a  part  of  the  early  design  activity,  documented 
in  Reference  1,  functional  requirements  were  devel- 
oped for  the  control  computer  hardware  to  be  used  in 
the  Sacramento  Control  Center.  These  requirements 
were  set  forth  in  a  request  for  proposal  (RFP)  which 
was  released  to  24  computer  manufacturers  and  sys- 
tems firms  in  September  1965  (Reference  2).  In  re- 
sponse to  this  RFP,  four  proposals  were  submitted. 
These  proposals  were  evaluated  by  both  the  Depart- 
ment and  the  cdnsultants  for  both  technical  conforma- 
tion to  the  RFP  and  cost.  The  results  of  this  evaluation 
indicated  a  lease  contract  should  be  awarded  to  the 
UNIVAC  Division  of  Sperry  Rand  Corporation  for  a 
Model  418-11  computer  system. 


Computer  System  Programming 

Since  the  Control  System  Model  study  was  a  "re- 
search and  development"  type  of  activity,  it  was  decid- 
ed that  the  programming  of  the  control  computer 
should  be  accomplished  "in-house"  by  a  team  of  engi- 
neers and  programmers  rather  than  to  contract  this 
activity  outside  the  Department.  The  Department  be- 


lieved it  essential  to  develop  this  capability  within  its 
own  staff  since  this  expertise  undoubtedly  would  be 
necessary  for  future  control  system  activities.  The 
programming  consisted  of  developing  software  for 
both  a  complete,  computer-controlled,  automatic  op- 
eration of  the  Aqueduct  and  an  operator-guide  opera- 
tion. This  programming  was  accomplished  between 
early  1966  and  1969. 

The  general-purpose  computer  software,  such  as 
the  executive  routine,  the  assembler,  and  other  utility 
programs,  was  supplied  by  UNIVAC.  The  programs 
needed  to  operate  the  Aqueduct  and  to  interface  with 
the  UNIVAC  software  were  written  by  a  departmen- 
tal team  including  engineers  with  civil,  electronic,  or 
electrical  backgrounds  and  computer  programmers. 
The  programs  were  written  in  assembly  language. 

Programs  were  written  for  two  modes  of  computer 
operation:  (1)  operator  guide;  and  (2)  real-time, 
on-line,  computer  control.  Both  of  these  modes  were 
tested  on  the  Model  to  determine  their  applicability  to 
the  main  line  (California)  Aqueduct  operation. 

The  plan  for  operator  guide  involved  having  the 
computer  monitor  and  analyze  different  operating 
situations.  The  computer  received  data  and  status 
from  each  of  the  sites,  analyzed  this  information,  and 
typed  suggested  advice  to  the  operator.  The  operator 
was  free  to  accept,  modify,  or  reject  the  advice. 

On  the  other  hand,  the  plan  for  real-time,  on-line, 
computer  control  provided  complete  command  and 
control  capability  from  the  computer.  The  computer 
would  perform  the  same  monitoring  and  analysis 
functions  as  in  the  operator  guide  but,  in  addition, 
automatically  form  and  send  the  communication 
messages  necessary  to  control  the  facilities. 

The  various  programs  required  for  computer 
control  of  the  Aqueduct  were  divided  into  six 
categories.  Specific  details  on  these  categories  and  the 
associated  computer  software  are  beyond  the  scope  of 
this  report.  However,  a  general  explanation  of  some  of 
the  programming  philosophies  follows: 

Master  Control  Program.  This  program  moni- 
tored and  directed  the  lower-level  application  pro- 
gram. It  selected  the  proper  program  at  the  proper 
time,  utilizing  the  real-time  clock,  and  determined  the 
sequence  of  execution  by  a  priority  system. 

Water  Delivery  Schedule  Program.  From  the 
available  water,  pumping  capacity,  and  water  delivery 
requests,  a  schedule  was  developed  of  the  amount  of 
water  to  be  delivered  at  each  turnout  at  the  designated 
time. 

Because  of  the  small  number  of  water  users  and  the 
absence  of  electrical  power  generation  or  water 
availability  limitations,  the  water  delivery  program 
for  the  South  Bay  Aqueduct  was  relatively  simple. 

Information  Control  Program.  This  category 
contained  the  "bookkeeping"  programs.  Status  of  all 
project  facilities  and  related  data  were  monitored  by 
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these  programs.  Information  was  kept  up  to  date  and 
available  to  programs  in  other  categories. 

The  information  storage  consisted  of  a  number  of 
sublevel  routines  which  accomplished  bookkeeping 
tasks  that  were  specifically  oriented  to  aqueduct 
operation.  A  main  function  was  "data/status  storage". 
When  data  or  status  messages  were  received  by  the 
computer,  this  routine,  among  other  things,  checked 
the  message  for  parity  and  activated  the  data/status 
storage  routine.  Then  the  message  was  repacked  into 
a  designated  storage  area  known  as  the  "data/status 
table".  When  the  new  information  was  stored,  its  time 
of  reception  was  also  placed  in  an  adjacent  storage 
location  thereby  maintaining  an  up-to-date  storage  of 
information  regarding  project  data  and  status. 

Daily  Operations  Program.  This  category  of 
programs  generated  a  plan  for  normal  operation, 
giving  the  day's  entire  flow  requirements  for  all  check 
structures,  pumping  plants,  and  turnouts.  If  the 
customer's  water  delivery  requests  changed  during 
the  day,  this  program  would  recompute  new  flow 
requirements  to  meet  the  change. 

The  daily  operation  plan  used  the  water  delivery 
schedule  and  data/status  tables  to  derive  a  daily  opera- 
tions plan  table.  The  plan  table  was  a  list  of  all  re- 
quired individual  site  activities  and  when  they  were  to 
be  performed  in  order  to  satisfy  the  water  delivery 
schedule.  The  plan  used  sophisticated  control  al- 
gorithms in  order  to  minimize  an\'  possible  hydraulic 
surges  due  to  a  planned  flow  modification. 

System  Operations  Program.  From  the  daily  plan 
and  flow  requirements,  this  program  selected  the 
proper  check  and  positioned  its  gates,  selected  and 
controlled  the  pumping  units,  and  opened  or  closed 
the  appropriate  turnouts.  This  program  performed 
the  operation  of  actual  water  delivery  to  customers. 

The  programs  in  this  category  were  designed  to 
route  all  commands  through  a  jurisdictional  control 
software  matri.x.  If  the  operational  mode  was 
computer  control,  the  command  was  permitted  to 
leave  the  computer  for  the  designated  site.  If  the  mode 
was  operator  guide,  the  command  was  typed  out  to  the 
operator  for  execution. 

Systems  Alarm  Program.  Responses  to  all  abnor- 
mal and  emergency  conditions  were  responded  to  by 
this  category  of  programs.  There  were  two  types  of 
alarms  involved:  (1)  those  that  came  from  instrumen- 
tation and  equipment  at  the  sites,  and  (2)  those  that 
were  generated  within  the  computer  by  analysis  of  the 


data  it  received.  At  the  time  an  analysis  was  being 
performed,  these  programs  called  on  other  categories 
to  supply  current  conditions  of  project  facilities  and, 
if  necessary,  called  upon  another  section  to  initiate  a 
command.  The  alarm  condition  might  also  require 
that  the  daily  operations  plans  be  rerun  because  the 
system  could  not  carry  out  the  previously  generated 
operations  plan. 

Results  from  the  Model  Study 

Although  many  problems,  both  developmental  and 
technical,  combined  to  delay  completion  of  the  test 
and  evaluation  of  the  South  Bay  Model  until  1970, 
many  benefits  accrued  prior  to  completion  to  make 
the  program  of  great  benefit  to  the  Department's  total 
control  system  effort.  These  benefits  were  realized 
concurrently  with  the  planning,  design,  and 
construction  efforts,  and  their  impact  was  felt  on 
other  control  system  activities  even  before  the  Model 
was  completed. 

Experience  gained  on  the  Model  was  invaluable  in 
the  design  and  construction  of  the  remainder  of  the 
project  control  system.  Many  technical  problems 
which  were  discovered  on  the  South  Bay  System  were 
avoided  on  the  main-line  control  svstem.  Certain 
organizational  changes  also  were  made  within  the 
Department  relative  to  the  design  and  construction  of 
aqueduct  control  systems.  These  changes  enhanced 
subsequent  control  system  activities. 

The  feasibility  of  using  computers,  both  as  building 
blocks  in  system  design  and  in  an  automatic  control 
environment,  was  firmly  established. 

The  consultants'  participation  was  invaluable  in 
assisting  the  Department  in  gaining  experience 
during  the  design  and  construction  of  the  Model. 
This,  in  turn,  allowed  the  Department  to  proceed 
with  the  implementation  of  the  main  control  system 
without  the  use  of  consulting  engineering  firms  as  had 
been  planned  for  Phases  2  and  i.  Consulting  assistance 
was  not  proposed  in  Phase  4. 

All  of  the  equipment  designed  and  constructed 
during  the  Model  study  is  now  being  utilized.  The 
South  Bay  Aqueduct  system  is  in  continuous  use  in 
the  operation  of  the  South  Bay  Aqueduct.  The 
Sacramento  Control  Center  equipment  has  been 
relocated  and  converted  to  a  control  simulator  for 
training  of  hydroelectric  plant  operators.  The 
UNIX'AC  computer  has  been  expanded  and  forms  the 
foundation  for  the  Project  Operations  Control  Center. 
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CHAPTER  IV.     OPERATIONS  CONTROL  PLAN 


During  construction  of  the  South  Bay  Model,  it 
became  evident  that  the  results  of  the  Model  were  not 
going  to  be  available  until  after  the  time  that  design 
of  the  remainder  of  the  Project's  control  systems  had 
to  be  started. 

Thus,  the  Operations  Control  Plan  for  the  State 
Water  Project  was  developed  to  provide  the 
preliminary  criteria  and  guidelines  by  which  the 
control  system  would  be  designed  and  constructed. 
This  plan  (completed  in  1965)  was  jointly  developed 
by  members  of  the  design,  construction,  and 
operations  and  maintenance  staffs. 

The  purpose  of  the  Plan  was  fourfold:  (1)  reassess 
the  overall  guidelines,  criteria,  and  basic  control  con- 
cepts for  the  project  control  system  as  originally  con- 
ceived and  discussed  in  Chapter  II,  "Planning 
Considerations";  (2)  furnish  information  about  the 
control  system  for  use  in  refinement  of  operation  and 
maintenance  activation  work,  such  as  manpower  re- 
quirements, organization,  and  operating  procedures; 
(3)  provide  a  definition  of  the  control  system  for  use 
as  needed  by  all  offices  in  the  Department;  and  (4) 
provide  the  control  system  interrelationship  between 
facilities  of  all  divisions  of  the  Project. 

Control  System  Plan 

The  Operations  Control  Plan  as  developed  defined 
the  structure  of  the  control  system  and  the  operational 
relationships  between  the  individual  facilities  to  be 
controlled.  The  Plan  requires  a  remote  manual 
control  system  with  expansion  capability  to  a  fully 
automated  system. 

Control  System  Structure 

The  control  system  structure  consists  of  five  control 
areas  which  are  located  geographically  to  encompass 
all  project  facilities.  An  area  control  center  is  located 
within  each  of  the  five  control  areas  and  is  the  normal 
point  of  control  for  all  facilities  within  that  area. 

A  Project  Operations  Control  Center,  located  in 
Sacramento,  is  responsible  for  coordinated  operation 
and  jurisdictional  control  over  the  entire  State  Water 
Project. 

Control  Areas 

The  five  control  areas  of  the  State  Water  Project  are 
Oroville,  Delta,  San  Luis,  San  Joaquin,  and  Southern 
California.  Figure  9  shows  the  boundaries  of  the  five 
control  areas. 

Oroville.  This  control  area  encompasses  the  entire 
Oroville-Thermalito  complex  and  the  Upper  Feather 
River  facilities,  with  an  area  control  center  located  at 
the  Oroville  Switchyard. 

Delta.  This  control  area  encompasses  the  Periph- 
eral Canal,  the  North  Bay  and  South  Bay  Aqueducts, 


and  the  North  San  Joaquin  Division  of  the  California 
Aqueduct.  Facilities  within  this  area  are  controlled 
remotely  from  the  Delta  Area  Control  Center  located 
in  the  Delta  Operations  and  Maintenance  Center. 

San  Luis.  This  control  area  encompasses  the  San 
Luis  Division  of  the  California  Aqueduct  and  includes 
San  Luis  Dam  and  Reservoir  (joint  facilities  of  the 
Department  and  the  U.  S.  Bureau  of  Reclamation). 
Facilities  within  this  area  are  controlled  remotely 
from  the  San  Luis  Area  Control  Center  located  in  the 
San  Luis  Pumping-Generating  Plant  building. 

San  Joaquin.  This  control  area  encompasses  the 
South  San  Joaquin  and  Tehachapi  Divisions  of  the 
California  Aqueduct  from  Kettleman  City  to  the 
Tehachapi  Afterbay  and  includes  the  Coastal  Branch. 
Facilities  within  this  area  are  controlled  remotely 
from  the  San  Joaquin  Area  Control  Center  located  in 
the  San  Joaquin  Operations  and  Maintenance  Center. 

Southern  California.  This  control  area  encom- 
passes the  East  Branch  Aqueduct  from  Tehachapi  Af- 
terbay through  Lake  Perris  and  the  West  Branch 
Aqueduct.  Facilities  within  this  area  are  controlled 
remotely  from  the  Southern  California  Area  Control 
Center  located  in  the  Southern  California  Operations 
and  Maintenance  Center  adjacent  to  Castaic  Dam. 


Operation,  Measurement,  Control,  and  Display 
Requirements 

Project  operation  requirements  were  analyzed,  and 
basic  measurement,  control,  and  display  requirements 
were  established  for  the  area  control  centers  and  for 
aqueduct  and  plant  facilities. 

Within  each  of  the  control  areas  of  the  State  Water 
Project,  capability  for  complete  remote  manual 
control  is  provided  from  a  control  console  located  in 
each  area  control  center.  Capability  for  control  of  each 
individual  aqueduct  site  or  plant  is  provided  locally 
from  a  site  control  panel  for  aqueduct  facilities  or 
from  a  plant  control  room  for  pumping  and  power 
plants.  In  addition,  at  plants,  a  level  of  control  is 
provided  from  the  unit  control  boards.  A  complete 
outline  of  the  specific  requirements  can  be  found  in 
Reference  3. 

A  general  summary  of  the  capabilities  from  each 
level  of  control  follows: 

Unit  Control  Boards— Plants.  The  unit  control 
board  provides  for  operation  of  pumping  and  generat- 
ing units  and  all  pertinent  auxiliary  equipment  with- 
out depending  on  any  control  signals  from  either  the 
plant  control  room  or  the  area  control  center.  The 
plant  operator  is  provided  with  all  monitoring  capa- 
bility required  to  ensure  proper  performance  of  the 
unit.  Displays  are  also  provided  by  indicating  meters. 
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Plant  Control  Room.  The  plant  control  room 
consolidates  the  operation  of  all  unit,  switchyard, 
station  service,  and  plant  auxiliary  functions  from  a 
single  location  within  the  plant.  This  control  is 
accomplished  without  dependence  on  any  signals 
from  the  area  control  center. 

The  plant  control  equipment  provides  complete 
startup  and  shutdown  sequencing  of  the  units  with 
monitoring  and  alarming  capability.  Alarms  and 
events  are  automatically  recorded.  Displays  consist  of 
indicating  displays  and  selected  recording. 

Local  Check  Structure  Control.  The  local  check 
structure  control  from  the  control  building  adjacent 
to  the  check  structure  provides  capability  to  monitor 
and  control  all  functions  at  a  check  structure.  Gate 
positions  are  controlled  and  monitored,  and  water 
levels  are  monitored  and  recorded. 

Local,  automatic,  water-level  control  capability  is 
required  whereby  gate  positions  are  automatically 
controlled  to  maintain  a  given  water-level  setpoint 
either  upstream  or  downstream  of  the  check  structure. 

All  control  is  possible  without  dependence  on  any 
signals  from  the  area  control  center. 

Turnout  Control.  Major  turnouts  are  provided 
with  both  monitoring  and  control  capabilities.  A 
major  turnout  is  one  with  a  design  capacity  greater 
than  5%  of  the  capacity  of  the  aqueduct  reach  in 
which  it  is  located,  or  200  cubic  feet  per 
second, whichever  is  less.  Minor  turnouts  require 
monitoring  capability  only. 

Other  Site  Control.  Other  facilities,  including 
dams,  reservoirs,  flow  measuring  stations,  and  water 
quality  monitors,  may  require  monitoring  and/or 
control  capability  locally  and,  in  some  cases,  from  the 
area  control  center.  As  the  need  arises,  such  capability 
will  be  incorporated. 

Area  Control  Center.  The  area  control  center 
consolidates  operation  of  all  remotely  controlled  sites 
within  the  area.  It  is  not  provided  with  capability  to 
monitor  or  control  facilities  of  another  area. 

The  area  control  center  communicates  through  lo- 
cal plant  control  systems  or  other  aqueduct  local  sys- 
tems to  provide  control  capability.  Data  monitoring  is 
provided  to  the  level  necessary  to  ensure  proper  oper- 
ation and  surveillance  of  all  facilities. 

Project  Operations  Control  Center 

The  Project  Operations  Control  Center  (POCC), 
located  in  the  Resources  Building  in  Sacramento,  is 
provided  with  capability  for  complete  monitoring  and 
dispatching  of  project  operation,  complete  jurisdic- 
tional control,  expansion  capability  to  full  real-time 
computer  control,  and  back-up  control  for  any  aque- 
duct area  control  center. 


This  control  center  monitors  area  operations, 
records  water  and  power  data,  develops  overall  svstem 
water  and  power  schedules,  optimizes  water  and 
power  allocations  for  the  Project,  and  coordinates 
interarea  functions. 

To  accomplish  dispatching  and  coordinated 
operation,  the  POCC  is  provided  with  the  capability 
to  monitor  and  display  the  same  functions  as  the  area 
control  centers.  Operation  of  the  POCC  is  not 
dependent  on  operation  of  any  area  control  center 
equipment.  Information  concerning  actual  project 
conditions  is  transmitted  directly  from  each  remote 
site,  thus  bypassing  the  area  control  center. 

Control  and  display  capabilities  necessary  to  assume 
control  of  any  one  of  the  aqueduct  area  control  centers 
are  provided.  Direct  control  capability  of  all  area 
facilities  can  be  accomplished  without  going  through, 
or  depending  upon,  any  equipment  in  the  area  control 
centers. 

Preparation  of  water  and  power  schedules  is  done 
by  an  on-line  computer  located  in  the  POCC.  This 
computer  will  automatically  compute  all  schedules 
and  transmit  these  schedules  to  the  area  control 
centers  through  remote  input-output  devices.  This 
computer  system  acts  as  the  control  point  from  which 
emergency  operation  of  the  Project  will  be  conducted. 

Communications  System 

Communications  are  required  to  tie  the  remote 
aqueduct  sites  to  their  area  control  centers  and  the 
area  control  centers  to  the  POCC.  Two  types  of 
communications  are  required,  data  and  voice. 

Data  Communications.  Data  communications  are 
required  to  provide  information  exchange  between 
remote  aqueduct  sites  and  area  control  centers.  Also 
included  are  data  communication  requirements  from 
the  area  control  centers  to  the  POCC  for  both  data  and 
the  computer's  remote  output  devices. 

Speed  requirements  for  data  communications  are 
dictated  by  the  need  to  control  the  facilities  to  their 
required  precisions  and  permit  complete  updating  of 
all  information  from  an  area  within  a  two-minute 
period. 

Extra  capacity  is  provided  in  the  initial  primary 
communications  along  the  Aqueduct  to  allow  for 
future  expansion  of  the  system. 

Voice  and  Teletype  Communications.  In 
addition  to  data  communications,  certain  voice  and 
teletype  communications  are  required  for  information 
exchange  betueen  various  points  uithin  the  Project. 
These  communications  are  carried  out  between  each 
remote  site  and  the  area  control  centers,  between  the 
area  control  centers  and  the  POCC,  and  between  the 
POCC  and  other  operating  agencies. 
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CHAPTER  V.     CONTROL  SYSTEM  DESIGN 


Following  completion  of  the  design  of  the  South 
Bay  Aqueduct  Control  System  Model  facilities  in 
1966,  the  plan  and  schedule  were  reviewed  for  design 
and  construction  of  the  California  Aqueduct  Control 
System  and  the  control  systems  for  the  Project's 
plants  and  the  Oroville-Thermalito  complex. 

It  became  apparent  that  if  control  systems  for  the 
Aqueduct  and  plants  were  to  be  in  operation  near  the 
time  the  facilities  themselves  were  to  be  placed  in 
operation,  the  Department  could  not  wait  for  the 
entire  test  and  evaluation  period  for  the  South  Bay 
Model  to  be  completed.  It  was  deemed  essential  that 
control  of  facilities  be  implemented  as  near  as  possible 
to  the  time  they  were  placed  in  operation  in  order  to 
minimize  operational  costs  and  operational  errors. 

Thus,  the  Department  embarked  on  the  design  of  a 
remote  manual  control  system  which  would  have  the 
capability  for  expansion  to  a  fully  automated  system 
in  conformance  with  the  operations  control  plan. 

To  ensure  compatibility  between  the  various 
designs,  certain  criteria  and  mandatory  specification 
requirements  were  developed  prior  to  the  start  of  the 
major  design  activities. 

Control  Systems  Consulting  Board 

As  discussed  in  Chapter  II,  as  of  1962  Phases  2  and 
}  contemplated  the  use  of  consulting  engineering 
firms  to  assist  the  Department  in  preliminary  systems 
design,  final  design,  and  development  of  contract 
specifications.  By  1966,  however,  the  Department  had 
developed  the  necessary  expertise  to  design,  construct, 
and  place  in  operation  the  various  control  systems. 
Nevertheless,  it  was  deemed  desirable  to  provide 
additional  expertise  to  review  the  staffs  work  to 
ensure  that  the  full  spectrum  of  technological 
information  was  available  for  the  system  designs. 

Accordingly,  in  September  1966,  the  Department 
retained  a  board  of  consulting  engineers  to  provide 
overall  guidance  during  the  implementation  of  the 
project  control  system. 

1  hree  consultants  with  experience  in  the  fields  of 
communications,  computers,  and  control  systems 
were  retained  as  the  Control  Systems  Consulting 
Board.  Mr.  Russell  G.  Hornberger,  (deceased) 
Chairman  of  the  Board,  had  long  experience  in  the 
design,  construction,  and  operation  of  large-scale 
plant  and  aqueduct  facilities.  Mr.  John  T.  Clabby 
(deceased)  was  a  recognized  expert  in  the  field  of 
computers  and  control  systems.  Mr.  Edward  W. 
Messinger,  Jr.  is  a  recognized  expert  in  the  field  of 
communications  engineering.  Upon  the  death  of  Mr. 
Clabby,  Mr.  Theodor  H.  Braun  was  retained  as  a 
member  of  the  Board.  Mr.  Braun  is  an  expert  in  the 
field  of  computers  and  software  development. 

The  Control  Systems  Consulting  Board  continued 
its  role  of  providing  general  technical  guidance  to  the 


Department  until  major  system  design  had  been 
completed.  The  Board  was  dissolved  in  June  1973, 
after  having  served  for  nearly  seven  years. 

System  Implementation  Plan 

Since  the  decision  had  been  made  to  have  the 
various  control  systems  made  operational  as  near  as 
possible  to  the  time  the  associated  project  facilities 
became  operational,  it  became  necessary  to  define  the 
method  by  which  all  systems  would  be  implemented. 
A  single  contract  in  1966-67  would  have  represented 
a  capital  investment  of  from  20  to  .30  million  dollars, 
with  a  construction  period  stretching  over  as  much  as 
10  years.  A  single  contract  of  this  magnitude  over  such 
a  long  period  of  time  would  have  presented  design  and 
contract  administration  difficulties  and  would  have 
precluded  state-of-the-art  advances  in  control  system 
equipment  from  being  utilized  in  the  Project's  control 
systems  as  they  became  operational. 

Multiple  contracts,  on  the  other  hand,  presented 
other  difficulties.  Since  all  the  various  systems 
ultimately  would  have  to  be  integrated  into  a  single 
projectwide  system,  they  all  had  to  be  compatible  in 
certain  areas  of  system  design  in  addition  to  being 
functionally  similar.  Multiple  contracts  also  had  a 
long-term  operation  and  maintenace  disadvantage  in 
that  there  could  be  no  assurance  of  like  equipment 
between  the  various  contracts. 

All  factors  considered,  the  decision  was  made  to 
implement  the  control  systems  with  a  series  of  con- 
tracts, with  the  Department  assuming  responsibility 
for  compatibility  between  the  various  systems.  This 
decision  dictated  the  requirement  for  certain  uniform- 
ity between  the  various  contracts. 

Design  Requirements 

Design  Criteria 

Detailed  criteria  for  the  design  of  the  control 
systems  were  developed  as  a  joint  effort  of  the 
operations  and  design  staffs  of  the  Department.  The 
requirements  of  the  systems,  including  the  structure 
and  the  functions  to  be  monitored  and  controlled  at 
the  various  facilities  (aqueducts,  plants,  area  control 
centers,  and  the  Project  Operations  Control  Center), 
were  contained  in  the  "Operations  Control  Plan" 
(Reference  3).  These  requirements  were  jointly 
reviewed  by  operations  and  design  personnel  as  to 
their  effect  on  the  designer's  operational  concepts  and 
as  to  the  availability,  feasibility,  and  economics  of 
equipment  to  satisfy  the  requirements. 

Final  design  criteria  consisted  of  the  control 
functions,  alarms,  data,  and  status  w  hich  were  to  be 
included  at  each  structure  and  control  center.  Where 
appropriate,  characteristics,  such  as  range,  accuracy, 
precision,  timing,  and  other  details,  were  included. 
These  criteria  were  organized  to  specify  the  monitor 
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and  control  requirements  locally  (at  the  structure  or 
device  controlled),  at  the  local  control  station  (e.g., 
inside  the  control  building  at  a  check  structure),  and 
at  the  control  centers. 

Although  there  have  been  additions  to  the  control 
requirements,  the  original  criteria  are  still  the  bulk  of 
those  actually  being  used  in  the  control  system. 

The  design  criteria  also  dictated  much  of  the  control 
system  design  by  generating  such  requirements  as 
speed  of  control,  frequency  of  monitoring,  number  of 
points  to  be  controlled,  and  degree  of  accuracy  and 
precision  required  for  measured  variables. 

Communication  Philosophy 

A  mandatory  design  requirement  for  all  systems 
was  the  method  of  communication  to  be  utilized 
between  the  remote  sites  and  the  control  centers.  This 
communication  was  selected  to  be  serial  digital  in 
format  to  meet  the  high  degree  of  accuracy  sought,  the 
large  number  of  points  to  be  monitored,  and  the 
problems  created  by  the  large  distances  involved. 

Digital  data  handling  has  some  disadvantages  when 
compared  to  analog  data.  During  the  design  period  in 
1966-67,  digital  data  handling  was  an  intrinsically 
more  expensive  approach,  particularly  where  the 
number  of  data  points  at  one  location  was  small.  It  also 
generally  required  higher  transmission  speeds  for  a 
given  data  updating  rate  because  all  data  had  to  be 
time  shared  on  a  communication  line. 

Digital  data  handling  has  two  major  advantages, 
however.  It  permits  a  greater  number  of  significant 
digits  in  a  measurement,  and  it  does  not  lose  accuracy 
with  transmission  to  distant  points.  These  two 
advantages  proved  decisive  in  view  of  the  design 
criteria  supplied. 

Serial  digital  communications  made  it  possible  to 
serve  all  sites  and  all  data  points  within  a  control  area 
on  no  more  than  two  sets  of  communication  lines. 
This  was  based  upon  being  able  to  monitor  1,000  data 
points  within  a  two-minute  period  on  a  single  set  of 
lines. 

Communication  Message  Format.  All  communi- 
cations between  facilities  are  accomplished  by  ex- 
change of  commands;  interrogations;  and  data,  status, 
and  alarm  messages. 

A  command  is  a  message  sent  from  a  control  center 
to  a  remote  site  and  contains  control  information, 
either  in  the  form  of  a  setpoint  (gate  position,  water 
level,  and  so  forth)  or  a  command  action  like  start 
pump. 

An  interrogation  is  a  message  sent  from  a  control 
center  to  a  remote  site  as  a  request  for  the  site  to 
respond  with  a  data,  status,  or  alarm  message. 

Data,  status,  and  alarm  messages  are  sent  from  a 
remote  site  to  the  control  center  and  contain 
quantitative  measurements  of  parameters  (data), 
conditions  such  as  units  in  service,  switchyard 
breakers  closed  (status),  and  alarm  conditions. 


The  method  and  arrangement  of  the  various  bits 
(binary  digits)  within  the  coded  message  allow 
information  to  be  transferred  and  be  intelligible  after 
the  transfer  has  been  completed.  Prior  to  design  of  the 
message  format,  it  was  necessary  to  determine  the 
extent  of  future  project  requirements  to  ensure  that 
the  system  would  be  capable  of  handling  any  possible 
future  expansion.  The  message  format  which  evolved 
after  considerable  study  was  a  36-bit  digital  "word". 

The  message  format  was  designed  with  sufficient 
bits  to  identify  all  the  sites  which  could  be  connected 
to  a  communication  line,  all  the  information  points 
which  could  occur  at  one  site,  and  also  to  provide  an 
information  portion  of  13  bits  for  quantitative  data 
pertinent  to  each  point.  For  status  and  alarm 
messages,  the  information  portion  can  carry  13  alarm 
conditions  or  13  status  points  per  message.  For  data 
messages,  the  information  portion  can  contain  an 
unsealed  decimal  value  of  4,095  plus  sign,  or  an 
unsealed  decimal  value  of  8,191  without  sign.  Three 
additional  bits  are  reserved  for  message  security, 
which  protect  against  communication  line  noise. 

The  other  essential  part  of  the  communication 
message  is  a  6-bit,  beginning-of-message  (sync)  code. 
This  part  has  a  fixed  format  so  that  it  is  always 
recognized  as  the  beginning  of  a  possible  message. 
Figure  10  shows  the  format  of  the  various  types  of 
communication  messages. 

Communication  Transmission  Rate.  All  com- 
munication messages  are  transmitted  at  a  rate  of  1,200 
bits  per  second.  This  rate  was  chosen  to  provide  a 
1,000  point  per  communication  line,  two-minute,  data 
update  rate.  Approximately  120  milliseconds  are  re- 
quired for  each  message  exchange  to  be  completed;  a 
total  of  60  milliseconds  to  clock  the  message  in  and  out 
of  the  remote  site,  12  milliseconds  site  response  time, 
and  an  average  of  48  milliseconds  for  a  two-way  com- 
munication line  propagation  delay. 

Standard  "data-sets"  or  "modems"  were  also 
available  at  this  speed.  Leased  telephone  lines  capable 
of  operating  at  this  speed  with  acceptable  error  rates 
were  readily  available  and  of  proven  reliability. 
Studies  showed  that  bit  rates  faster  than  1,200  would 
reduce  reliability  and  increase  costs  without  a 
commensurate  increase  in  the  capacity  of  a  line. 

Communication  Protocol.  The  communications 
system  is  operated  in  a  "polled"  environment.  In  this 
environment,  each  data,  status,  or  alarm  message  is 
interrogated  from  the  control  center  and  the  answer 
received  before  the  control  center  issues  the  next 
interrogation.  If,  for  some  reason,  the  interrogation  is 
not  answered,  the  next  interrogation  will  be  sent  after 
a  suitable  delay  period  (usually  about  100-200 
milliseconds). 

A  polling  technique  was  chosen  to  allow  flexibility 
to  alter  the  interrogation  scan  sequence  or  to 
interrogate  selected  points  more  frequently  than  once 
every  two  minutes. 
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Redundancy  of  Communications.  Another  de- 
sign feature  uas  the  provision  for  baci<-up  communi- 
cations to  each  major  site  from  the  control  center. 
This  is  accomphshed  by  approaching  each  site  with 
communication  lines  from  two  geographically  sepa- 
rate routes.  A  primary  or  "A"  line  is  routed  to  each 
site  in  turn  from  the  control  center  by  the  most  direct 
route,  usually  along  the  Aqueduct.  A  back-up  or  "B" 
line  is  routed  to  the  end  of  the  aqueduct  section  and, 
starting  with  the  last  site,  comes  into  each  site  in  re- 
verse order  and  from  the  opposite  direction. 

A  more  detailed  description  of  the  actual  communi- 
cation line  routing  is  contained  in  Chapter  \'I1,  "Com- 
munication Systems". 

Security  of  Communication  Transmissions.  Al- 
though the  communication  lines  leased  from  tele- 
phone companies  are  reliable  and  a  high  degree  of 
availability  has  been  achieved,  they  are  still  subject  to 
certain  error-producing  phenomenon  and  cata- 
strophic failure.  The  most  prevalent  communications 
problem  is  that  of  impulse  noise. 

The  aqueduct  cable,  which  forms  the  entirety  of  the 
"A"  route  from  an  area  control  center  (ACC)  and  a 
substantial  portion  of  the  "B"  route,  is  almost  immune 
from  random  impulse  noise  because  it  does  not  pass  in 
pro.yimity  to  other  communications,  does  not  go 
through  telephone  exchanges,  and  is  shorter.  The  "B" 
route  suffers  from  random  impulse  noise,  which 
causes  some  message  failures  due  to  the  higher  "error 
rate".  This  effect  is  shown  by  more  frequent  single- 
message  failures  due  to  bits  in  the  message  being 
changed  by  impulse  noise.  Messages  lost  in  this  man- 
ner are  usually  recovered  on  the  next  interrogation 
scan  so  that  the  effect  of  impulse  noise  on  the  control 
system's  effectiveness  is  minimal.  Messages  altered  by 
impulse  noise  are  detected  by  a  parity  checking 
scheme  contained  in  the  message  format. 

Three  bits  of  parity  are  used  in  the  security  scheme. 
The  (P,)  parity  bit  is  odd  parity  on  bit  numbers 
(3N+1),  Pj  is  odd  parity  on  bit  numbers  (3N-I-2), 


bit   3n  +3j  n  =  9 


and  P,  is  odd  parity  on  bit  numbers  (3N-(-3),  where 
N  is  an  integer  from  1  to  9  (Figure  11).  Parity 
computation  is  not  required  on  the  6  bits  in  the  sync 
code.  This  cyclic  parity  scheme  protects  100%  against 
undetected  message  errors  resulting  from  3  adjacent 
bits  or  less  of  the  message  being  in  error.  Using  a 
communication  line  error  rate  of  1  in  10'  bits,  this 
parity  scheme  will  detect  message  errors  so  that  only 
\}5  out  of  10'"  messages  sent  will  contain  an 
undetected  error. 

Additional  security  techniques  have  been  applied  to 
command  messages  to  guard  against  the  possibility  of 
invalid  commands  causing  false  operations. 

Two  types  of  catastrophic  failures  which  are  most 
likely  to  occur  on  the  communication  lines  are  line 
breaks  and  induced,  high-amplitude,  uhite  noise 
(noise  which  contains  equal  distribution  of  frequen- 
cies over  the  entire  band  uidth  of  the  communication 
channel). 

With  dual-routed  facilities,  the  effect  of  a  single  line 
break  is  minimal  if  both  routes  are  in  continual  use 
(the  alternate  route  is  an  active  system) .  The  Project's 
data  communications  operate  in  an  active  mode. 
Interrogation  scans  from  the  ACC  or  POCC  are 
alternated  for  each  cycle  between  the  "A"  and  "B" 
routes.  Data  responses  from  the  sites  are  sent  back  to 
the  ACC  and  POCC  simultaneously  over  both  routes. 
This  method  takes  advantage  of  the  diversity  of  routes 
while  facilitating  detection  of  failed  communications. 
When  a  failure  of  one  interrogation  line  does  occur,  it 
does  not  make  the  system  inoperable  but  merely 
extends  the  updating  time  for  data  to  four  minutes 
instead  of  two.  Commands  are  always  sent  out 
sequentially  over  both  lines. 

The  length  of  time  required  for  an  interrogation 
and  its  response  is  a  known  value  and  varies  with  the 
distance  from  the  control  center  to  the  site  being  inter- 
rogated. This  time  averages  about  120  milliseconds.  If 
the  site  does  not  respond,  as  may  happen  if  the  com- 
munication lines  are  not  functioning  properly  or  the 
site  itself  is  inoperative,  the  control  center  waits  the 
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Figure   11.     Message    Security   Scheme 
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longest  expected  time  (about  200  milliseconds)  and 
then  proceeds  with  the  next  interrogation.  Failures  of 
this  kind  are  compiled  and  analyzed  by  the  control 
center  computer  to  determine  the  probable  cause  and 
inform  the  operator  of  this  determination.  Thus,  fail- 
ures can  be  immediately  detected  and  repair  work  can 
begin  even  though  valid  data  are  continually  being 
received. 

The  second  type  of  catastrophic  failure  (induced, 
high-amplitude,  white  noise)  may  obliterate  some  or 
all  actual  messages  or  may  simulate  a  message  where 
one  does  not  exist. 

Obliteration  of  a  small  fraction  of  messages  is  not  a 
serious  problem  since  it  does  not  result  in  erroneous 
data  but  only  a  delay  in  information  updating.  The 
problem  is  more  serious  when  the  number  of  messages 
obliterated  is  large  or  includes  all  the  data  to  or  from 
one  of  the  sites.  Simultaneous  noise  of  this  magnitude 
on  both  "A"  and  "B"  routes  is  extremely  rare. 

A  more  dangerous  and  less  anticipated  problem  is 
the  synthesis  of  messages  from  white  noise.  This  will 
occur  under  rather  special  conditions  where  ( 1 )  there 
is  no  signal  on  the  line,  (2)  the  noise  is  unusually  high 
in  amplitude,  and  (3)  the  modem  receivers  are 
adjusted  to  be  very  sensitive.  Under  these  conditions, 
random  messages  will  be  clocked  into  the  system  at  a 
rate  of  1,200  per  second.  Since  this  possibility  has  the 
greatest  potential  for  damage  when  the  simulated 
message  is  a  command,  the  preventive  measures  taken 
apply  to  commands. 

One  technique  is  a  reduction  in  the  sensitivity  of  the 
receivers  and  squelching  of  site  receiver  during 
periods  of  no  signal.  The  other  technique  to  counter 
the  synthesizing  of  commands  is  to  require  double 
transmission  of  commands  to  a  site  within  a  specified 
period  of  time  before  the  commanded  function  will  be 
performed.  This  measure  is  effective  because  the 
probability  of  synthesizing  two  identical  messages 
within  a  short  period  of  time  is  extremely  low. 

Standard  Communications  System.  Each  remote 
site  and  control  center  is  provided  with  a  standard 
communications  system  containing  modems  and 
special  communications  terminal  equipment.  These 
communication  systems  are  of  uniform  functional 
design  throughout  the  entire  Project. 

Nlodems  are  devices  which  convert  direct-current 
( DC)  electrical  pulses  into  appropriate  audio  frequen- 
cies (modulation)  for  transmission  on  a  voice-grade 
communication  line  and  convert  received  audio  fre- 
quencies into  direct-current  pulses  (demodulation). 
They  are  necessary  because  high-speed  DC  pulses  can- 
not be  transmitted  farther  than  a  few  hundred  feet 
over  a  pair  of  wires  without  becoming  unintelligible. 
The  modems  specified  for  the  control  system  are  com- 
patible with  Western  Electric  Company,  Model  202D. 
The  communications  terminal  equipment  (CTE)  is 
a  system  of  special  solid-state,  electronic,  logic  ele- 


ments which  interface  the  modems  to  the  remainder 
of  the  equipment  at  the  site  or  control  center.  For 
incoming  messages,  the  CTE  converts  the  serial  data 
stream  coming  from  the  modem  into  a  parallel  form 
suitable  for  the  computer  input.  It  also  checks  for  the 
presence  of  the  sync  code  at  the  beginning  of  a  mes- 
sage. The  CTE  contains  a  receive  clock  which  is  syn- 
chronized by  the  sync  code  bits.  This  clock  controls 
the  sampling  of  the  incoming  data.  Outgoing  messages 
are  serialized  by  the  CTE  and  clocked  into  the  modem. 
I  he  C  TE  also  exchanges  numerous  timing  and  con- 
trol signals,  both  with  the  computer  and  the  modems, 
to  maintain  synchronization,  to  isolate  the  me.ssages 
from  pulses  which  could  be  mistaken  for  bits,  and  to 
place  the  bits  in  their  proper  position  within  the  mes- 
sage. 

The  actual  interface  between  the  CTE  and  modems 
was  specified  to  be  in  conformance  with  EIA-RS-232B 
(Reference  4).  The  interface  between  the  CTE  and 
the  computer  was  left  to  the  discretion  of  the  system 
contractor. 

Equipment  and  System  Reliability 

Of  primary  concern  was  the  order  of  reliability  that 
could  be  economically  achieved  and  how  it  could  be 
accomplished.  It  was  realized  that  as  equipment  relia- 
bility was  increased  from  that  which  had  been  utilized 
for  many  years  in  the  water  utility  industry  to  that 
which  was  being  developed  currently  as  part  of  the 
space  age,  the  cost  of  the  system  would  increase  com- 
mensurately.  Another  concern  was  that  developments 
in  this  type  ot  equipment  were  being  made  almost 
daily  and,  since  control  system  construction  would 
span  many  years,  design  had  to  be  progressive  and 
farsighted  to  provide  for  the  unavoidable  obsolescense 
which  would  take  place  during  the  design  and  con- 
struction period. 

The  plan  which  was  utilized  consisted  of  specifying 
state-of-the-art  materials,  imposition  of  quality 
control  requirements,  system  availability  analysis, 
system  testing,  system  availability  demonstration,  and 
contractor  prequalification. 

State-of-the-Art  Materials.  Where  practical,  all 
components  were  specified  to  incorporate  semicon- 
ductor circuitry  throughout,  all  diodes  and  transistors 
were  to  be  of  the  silicon  type,  and  all  electronic  equip- 
ment was  to  incorporate  integrated  circuits. 

All  equipment  and  material  to  be  used  were  to  have 
been  in  current  manufacture  at  the  time  of  the  bid 
opening.  In  addition,  suppliers  of  equipment  were  to 
have  been  actively  engaged  in  the  manufacture  of 
similar  equipment  for  at  least  three  years  prior  to  the 
time  of  the  bid  opening. 

Quality  Control  Requirements.  All  contractors 
were  required  to  obtain  department  approval  on  all 
proposed  equipment  and  adhere  to  a  quality  control 
procedure.  This  procedure  provided  for  the  preven- 
tion and  ready  detection  of  discrepancies  and  for  time- 
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ly  and  positive  corrective  action.  All  phases  of  the 
contractor's  work,  whether  performed  within  the  con- 
tractor's facilities  or  at  other  sources,  had  to  be  con- 
trolled at  all  points  necessary  to  ensure  conformance 
with  contractual  requirements.  The  quality  control 
procedure  covered  control  of  purchases,  source  in- 
spection, in-manufacture  inspection,  test  and  check- 
out, and  final  inspections. 

System  Availability  Analysis.  Within  90  days  after 
receipt  of  the  notice  to  begin  work,  the  contractor  was 
required  to  submit  an  inherent  availability  analysis  of 
the  system  he  proposed  to  provide,  using  the 
following  availability  prediction  procedure. 


Ai  = 


(MTBF)i 


(MTBF),  +  (MTR)i 


X    100 


where  A;  =  Inherent  System  Availability  (percent) 
(MTBF)i  =  Predicted  Mean  Time  Between  System 

Failures  (arithmetic  mean) 
(MTR)j  =     Predicted  Mean  Time  to  Repair  a  Sys- 
tem Failure  (arithmetic  mean) 

Continuation  of  the  contract  was  contingent  upon  the 
contractor  successfully  meeting  a  minimum  specified 
availability  level.  All  equipment  which  the  contractor 
planned  to  utilize  as  part  of  his  installation  was 
monitored  prior  to  purchase  to  ascertain  that  the 
quality  was  commensurate  with  that  on  which  the 
above  calculation  was  based.  Materiel  was  rejected  or 
accepted  on  that  basis.  The  equipment  assembly  at  the 
contractor's  plant  was  closely  monitored  to  ensure 
that  the  most  modern  and  effective  procedures  were 
employed  and  that  the  assembly  was  in  conformance 
with  availability  calculations. 

System  Testing.  Extensive  testing  was  specified  at 
both  component  and  system  levels.  Each  component 
of  each  subsystem  required  testing  by  the  contractor. 
Where  duplication  of  components  occurred,  a  random 
sampling  technique  was  used  on  which  complete  tests 
of  no  less  than  10%  of  the  total  number  of  components 
installed  could  be  employed.  The  Department 
reserved  the  right  for  100%  testing  where  required. 
Additional  testing  was  required  where  failures 
occurred  in  the  random  sampling. 

Following  manufacture,  the  system  was  required  to 
be  assembled  at  the  contractor's  facility  for  a  final 
factory  acceptance  test.  This  test  included  complete 
operation  of  all  components  as  a  system  with 
simulated  inputs  and  outputs.  Upon  acceptance,  the 
system  was  disassembled  and  shipped  to  the  site  for 
installation. 

After  the  installation  work  was  completed,  the 
contractor  was  required  to  conduct  operational 
completion  tests  to  demonstrate  compliance  with  the 
operational  requirements  of  the  specifications. 

System  Availability  Demonstration.  Upon  com- 
pletion of  all  installation  and  testing,  the  Department 
took  possession  of  the  systems  for  use  during  an  opera- 


tional availability  demonstration.  During  this  f)eriod 
of  time,  which  could  vary  from  six  months  to  one  year, 
the  Department  used  the  systems  and  the  contractor 
maintained  all  the  equipment.  During  a  continuous 
six-month  period,  the  contractor  was  required  to  dem- 
onstrate that  the  system  met  certain  minimum  availa- 
bility levels.  Failure  records  were  kept  and  analyzed 
by  the  Department,  and  the  availability  was  calculated 
using  actual  MTBF  and  MTR  figures  rather  than  pre- 
dicted values.  If  the  required  level  of  availability  was 
not  achieved  within  a  period  of  one  year  after  being 
placed  in  operation,  the  contractor  would  be  assessed 
liquidated  damages,  which  provided  funds  for  neces- 
sary modifications  to  the  system. 

Naturally,  the  larger  the  system,  the  smaller  the 
required  availability  level  would  be,  due  to  the  greater 
number  of  components  where  failures  could  occur. 
Thus,  for  each  contract,  a  minimum  availability  level 
was  assigned  on  the  basis  of  the  total  number  of 
components  being  supplied  and  installed  in  that 
contract.  These  minimum  levels  were  in  the  range  of 
95%  for  the  aqueduct  systems  and  in  the  range  of  99% 
for  the  plants. 

The  specifications  limited  the  total  number  of  fail- 
ures which  would  be  allowed  during  the  availability 
demonstration.  It  was  anticipated  that  a  contractor 
might  overcome  the  quality  requirement  implied  by  a 
high  availability  requirement  by  the  process  of  rapid 
repair  of  failed  equipment  (short  MTR).  To  prevent- 
this  expedient,  a  minimum  mean-time-between-fail- 
ures  of  50  hours  was  specified. 

To  ensure  that  the  failure  characteristics  developed 
during  the  availability  demonstration  were  sufficient 
on  which  to  base  system  acceptance,  the  availability 
calculation  and  MTBF  rate  had  to  have  a  confidence 
limit  of  at  least  95%  using  the  chi-square  distribution 
with  the  following  equation. 


C,  = 


2r 


(MTBF) 


X'a:2r 

Where  Cl  =    One  sided  lower  confidence  limit 
r  =    Number  of  failures 
X^a:2r  =    Chi-squared  distribution  value  for  the 
percentage  point  corresponding  to  the 
specific  confidence  level  and  degrees  of 
freedom 
100(1— a)=    Confidence  level  (percent) 
2r  =    Degrees  of  freedom 
a  =    Probability  that  chi-squared  value  will 
be  exceeded 

Contractor  Prequalification.  The  success  of  the 
procedures  and  techniques  developed  to  ensure  sys- 
tem reliability  were  dependent  on  the  contractors 
who  were  required  to  adhere  to  them.  Consequently, 
contractor  and  subcontractor  prequalification  was  re- 
quired to  limit  contract  awards  only  to  those 
experienced  in  this  kind  of  control  system  implemen- 
tation. 
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These  prequalification  procedures  are  discussed  in 
Chapter  VI,  "Control  System  Construction." 

Operation  and  Maintenance  Documentation 

A  key  requirement  for  all  system  specifications  was 
adequate  documentation  of  the  contractor's  hardware 
and  software. 

Hardware.  Contractors  were  required  to  provide 
the  Department  with  complete  instruction  manuals 
for  maintenance  of  the  equipment.  This  requirement 
included  both  a  system  instruction  manual  as  well  as 
various  equipment  manuals. 

The  system  instruction  manual  was  specified  to  be 
a  specially  written  document  to  convey  an 
understanding  of  how  the  system  operates  and  to 
provide  sufficient  procedures  for  operation  and 
maintenance.  This  manual  provides  instructions  for 
operation  and  maintenance  of  the  complete  system  as 
opposed  to  operation  and  maintenance  of  individual 
equipment  of  which  the  system  is  comprised. 

The  system  instruction  manual  is  organized  along 
the  following  lines: 

1.  Introduction — Contains  a  detailed  description 
of  equipment.  The  primary  purpose  of  the 
introduction  is  to  provide  a  ready  orientation  to  the 
use  and  purpose  of  the  manual  and  its  relationship  to 
the  equipment  manuals. 

The  extent,  content,  and  limitations  of  maintenance 
covered  by  the  manual  are  explained  in  this  section. 
The  section  also  includes  special  circumstances  and 
environmental  considerations,  if  applicable. 

2.  Safety  Precautions — Describes  all  major 
hazards  to  personnel  and  equipment  and  safety 
considerations  that  are  peculiar  to  the  equipment  or 
jobs  covered  in  the  manual.  The  information  is  in 
tabular  form  wherever  practical. 

3.  Description  of  System — Provides  a  physical 
and  functional  description  of  the  system  in  its 
operating  environment  but  does  not  describe  in  detail 
the  individual  equipment  in  the  system. 

4.  Physical  Description — Describes  the  physical 
features  of  the  system  and  their  general  arrangement. 
It  also  provides  a  brief  description  of  the  major 
components  of  the  system,  their  location,  and 
relationship  to  the  complete  system. 

5.  Functional  Description — Explains  how  the 
system  functions  as  an  integral  unit. 

6.  System  Operating  Instructions — Contains  the 
necessary  information  required  by  maintenance  per- 
sonnel for  operation  of  the  equipment  and  includes 
such  instructions  as  necessary  for  setting  up  or  pre- 
paring the  equipment  for  use,  warm-up  procedures  (if 
applicable),  starting  the  system,  verifying  normal  op- 
eration, shutdown,  postshutdown,  and  emergency 
procedures. 


Emergency  procedures  consist  of  actions  to  be  tak- 
en by  an  operator  or  maintenance  technician  in  the 
event  of  malfunction. 

Detailed  step-by-step  procedures,  or  specific  se- 
quence of  accomplishment,  if  required,  are  presented 
in  tabular  or  checklist  form  to  the  maximum  extent 
feasible. 

7.  System  Maintenance  Instructions— Contains 
the  applicable  checkout,  trouble  shooting,  servicing, 
remove  and  replace,  and  inplace-repair  procedures 
which  are  performed  on  a  system  basis. 

8.  Checkout  Procedures— Provides  checkout 
procedures  required  to  verify  the  satisfactory  opera- 
tion of  the  system  or  major  subsystem  as  applicable. 

9.  Trouble  Shooting  Procedures— Serves  as  a 
guide  in  isolating  faulty  equipment.  Also  indicates, 
but  does  not  describe  in  detail,  the  corrective  action 
required. 

10.  Servicing — Includes  cleaning,  lubrication,  re- 
plenishment of  fuel,  and  other  preventive  mainte- 
nance procedures  that  apply  to  the  particular 
equipment. 

11.  Remove  and  Replace  Procedures — Included 
only  when  they  are  not  provided  in  the  applicable 
equipment  manuals.  The  procedures  provide  step-by- 
step  instructions  for  removal  and  replacement  of 
items  which  are  subject  to  frequent  replacement. 

12.  Repair  Procedures — Provided  only  in  those 
instances  where  the  work  can  be  performed  in  place 
or  on  a  system  basis,  or  a  combination  of  these.  The 
procedures  provide  the  necessary  information  to 
bring  the  equipment  up  to  the  required  serviceable 
standard.  If  checkout  is  required  to  verify  satisfactory 
operation  of  the  equipment,  applicable  reference  is 
made  to  the  appropriate  section  of  the  manual. 
Reference  to  the  equipment  manuals  is  used  wherever 
possible. 

13.  Wiring  Diagrams — Includes  power  distribu- 
tion and  control  wiring  diagrams  with  associated  data 
required  to  understand  the  structure,  arrangement, 
and  function  of  the  system  (simplified  functional  or 
logic-type  diagrams  are  sometimes  used);  trace  cir- 
cuits; procedures  to  make  continuity  checks;  and  gen- 
eral and  specific  trouble  shooting  techniques  for 
inoperative  or  malfunctioning  circuits. 

Equipment  manuals,  on  the  other  hand,  consist  of 
an  organized  assembly  of  maintenance  and  operating 
manuals,  bulletins,  brochures,  and  other  data 
normally  furnished  by  the  manufacturer  with  his 
equipment.  Supplemental  data  are  included  to 
augment  that  normally  provided  by  the  manufacturer. 

These  equipment  maintenance  manuals  include 
information  and  data  in  sufficient  detail  to  permit  a 
trained  technician  to  troubleshoot,  diagnose,  and 
repair  the  control  equipment  and  peripherals 
including  the  power  supply  and  the  C'FE  equipment. 
The  following  specific  subjects  are  covered  in  detail: 
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1.  Description  of  equipment 

2.  Normal  operating  procedures 

3.  Emergency  operating  procedures 

4.  Preventive  maintenance 

5.  Calibration 

6.  Repair  and  replace  instructions 

7.  Complete  schematic  diagrams 

8.  Complete  wiring  diagrams 

9.  Parts  list  for  each  component  or  device, 
manufacturer's  stock  or  catalog  number,  quantities 
used,  descriptions,  and  illustrations. 

10.  Component  value  and  rating  designations  for 
all  components,  devices,  and  equipment  regardless  of 
whether  or  not  the  manufacturer's  stock  number  is 
furnished. 

11.  Recommended  spare  parts  list 

12.  Voltage  and  current  or  similar  test  point  data 

13.  Trouble-shooting  procedures  for  central 
processor  units  and  other  equipment. 

14.  Timing  diagrams  relating  to  control  and 
monitoring  under  program  control. 

Software.  The  contractors  also  were  required  to 
document  all  of  the  software  they  developed  for  the 
various  systems.  This  documentation  includes: 

1.  General  Description — Presents  the  overall 
purpose  of  the  computer  program.  All  assumptions  by 
the  contractors  are  stated  along  with  their  logic. 
Formats  for  the  input  and  output  messages  also  are 
described. 

2.  Abstracts — Describe  the  operational  objectives 
for  each  major  section  of  the  program  logic  and 
includes  the  relationships  to  other  major  sections. 

3.  Nondependent  Flow  Chart — Each  major  sec- 
tion of  the  programming  logic  is  presented  in  greater 
detail  than  for  minor  sections.  This  detail  is  developed 
into  a  format  of  flow  charts  using  statement  and  deci- 
sion blocks  with  lines  connecting  these  blocks  to  show 
the  flow  of  information.  Within  each  statement  and 
decision  block,  sufficient  information  is  presented  to 
describe  what  is  being  accomplished. 

4.  Dependent  Flow  Charts — Describe  the  non- 
dependent  flow  charts  in  such  detail  that  a  real-time 
programmer  can  directly  accomplish  the  required 
coding.  They  are  developed  in  a  flow-chart  format 
using  statement  and  decision  blocks  with  lines  con- 
necting the  blocks  to  show  the  flow  of  information. 
These  flow  charts  incorporate  the  capabilities  of  the 
specific  processor  and  programming  language  being 
employed. 

5.  Assembled  Listing — A  complete  and  fully 
debugged  assembled  listing  is  furnished  of  all  coding 
required  for  successful  execution  of  supplied  software 
programs. 

The  listing  includes  location  counter  values 
(P-register  values),  machine  language  operation 
codes,  machine  language  operands,  coding  labels, 
mnemonic  operation  codes,  mnemonic  operands,  and 
comments. 

6.  Memory  Map — A  map  is  provided  of  the 
internal  storage  of  the  central  processor  unit.  This 
map  shows  the  numbered  locations  of  the  core 
memory  and  the  information  stored  in  each  location. 


If  a  drum  or  disk-type  storage  device  is  used,  a  map 
of  the  device  is  supplied.  The  map  shows  the  memory 
location  and  the  information  stored  in  these  locations. 

7.  Glossary  of  Variables — A  glossary  listing  is 
included  for  every  variable  used  in  the  coding  with  a 
label  and  description  for  each  variable. 

Operation  and  Maintenance  Training 

The  contractors  were  required  to  develop  and 
conduct  special  training  courses  for  the  Department's 
operation  and  maintenance  personnel.  These  courses, 
averaging  from  one  to  two  months  in  length,  covered 
detailed  instructions  on  the  operation,  maintenance, 
adjustment,  calibration,  and  troubleshooting  of  the 
hardware  components. 

Software  training  also  was  required,  including 
overviews  and  details  of  the  software  design.  This 
training  averaged  from  two  to  four  weeks  in  duration. 

Design  of  Check  Structure  Control  Systems 

There  are  64  check  structures  along  the  California 
Aqueduct  of  the  State  Water  Project.  Their  purpose  is 
to  control  the  rate  of  flow  in  the  Aqueduct  by  move- 
ment of  the  radial  gates  to  vary  the  cross-sectional  area 
of  flow  through  the  check  structure  to  achieve  a  bal- 
ance between  flow  rate  and  upstream  and  down- 
stream water  level. 

These  check  structures  are  spaced  from  S  to  10  miles 
apart  along  the  Aqueduct,  depending  in  part  on  the 
design  capacity  of  the  reach  in  which  they  are  located. 
Each  check  structure  has  two,  three,  or  four  radial 
gates,  depending  on  its  location  in  the  system. 

The  gates  are  positioned  by  an  electrically  driven 
hoist  mechanism  which  is  cable-connected  to  the  gate 
itself.  The  hoisting  mechanism  is  controlled  on-site 
from  either  a  local  control  switch  adjacent  to  the  hoist 
on  the  deck  of  the  check  structure  or  from  a  motor 
control  center  located  within  the  control  building 
associated  with  each  check  structure.  Water  levels  and 
gate  positions  also  can  be  read  locally  by  means  of  staff 
gauges. 

The  equipment  added  for  control  consists  of  a 
computer  system;  an  operator's  control  panel; 
communications  equipment;  devices  for  transducing 
operating  parameters;  local,  automatic,  water-level 
control  equipment;  and  subordinate  site  and  interface 
equipment.  Figure  12  is  a  block  diagram  of  a  typical 
check  structure  control  system. 

Computer  System 

The  basic  element  of  the  control  system  at  the  check 
structures  is  a  small,  general-purpose,  digital  com- 
puter system.  This  computer  interfaces  with  the  com- 
munication terminal  equipment,  the  local  operator's 
control  panel,  the  various  measuring  and  control  de- 
vices, and  the  subordinate  site  equipment. 

Several  factors  pointed  to  the  possibility  that  a 
small,  general-purpose,  digital  computer  (the  type 
now  referred  to  as  a  "mini-computer")  might  be 
economically  justified  for  use  at  check  structure  sites. 
These  factors  were:  (1)  the  high  accuracies  required, 
such  as  1  part  in  300  for  gate  position  and  1  part  in 
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1,000  for  water  level;  (2)  the  need  to  encode  or  decode 
all  communication  setpoints  and  transmission 
measurements  into  a  binary  digital  code;  (3)  the  need 
for  dynamic  comparisons  and  control  actions,  such  as 
actual  gate  position  with  gate  position  setpoints;  (4) 
the  need  for  computational  capability;  and  (5)  the 
convenience  of  using  check  structures  as  gathering 
points  for  data  and  control  for  turnouts  and  other 
subordinate  sites. 

Studies  were  made  to  determine  the  relative  econ- 
omy and  reliability  of  systems  using  logic  circuits  spe- 
cifically designed  for  the  functions  to  be  performed 
versus  using  a  general-purpose  mini-computer.  The 
studies  did  not  provide  an  unequivocal  answer  to 
these  questions  but  did  indicate  that  a  mini-computer 
was  economically  competitive  at  the  prices  then  cur- 
rent and  was  equal  in  reliability. 

There  were  other  features  of  the  mini-computer 
which  shifted  the  balance  toward  its  use.  When  a  com- 
puter is  used  for  control,  changes  and  additions  can  be 
conveniently  made  to  the  computer  software  without 


Figure   13.      Operolor's    Control    Panel — Check    Siruclure    Conlrol    Syslen 


putting  the  controlled  equipment  out  of  service  or 
purchasing  additional  equipment.  Computer  changes 
and  additions  can  be  made  "off  line"  and  loaded  into 
the  site  computer  after  the  program  has  been  com- 
pleted and  tested.  When  a  computer  is  used,  the  re- 
quired hardware  is  simpler  and  more  readilv  available 
"off  the  shelf,  resulting  in  easier  maintenance  and 
replacement.  Proper  documentation  of  program 
changes  is  also  easier  to  ensure  than  is  documentation 
of  hardware  modifications. 

As  a  result  of  these  studies,  design  of  the  check 
structure  controls  incorporated  the  use  of  a 
general-purpose  digital  computer.  At  most  check 
structures,  this  computer  is  a  4K  core  memory 
computer.  The  computers  used  at  the  check  structures 
in  the  Delta,  San  Luis,  and  San  Joaquin  control  areas 
are  Hewlett  Packard  Model  2114.  Those  used  in  the 
Southern  California  control  area  are  Honeywell 
Model  316. 

Operator's  Control  Panel 

Each  check  structure  is  provided  with  local 
operator  control  capability  through  the  operator's 
control  panel.  This  panel  is  the  operator  interface  to 
the  computer  for  the  purpose  of  monitoring  and 
controlling  the  various  parameters  and  devices  at  the 
structure. 

The  panel  consists  of  a  series  of  switches, 
pushbuttons,  and  digital  readouts  with  which  the 
operator  communicates  with  the  svstem.  Figure  13 
shows  the  layout  of  a  typical  operator's  control  panel. 

Communications  Equipment 

The  communications  equipment  consists  of  the 
standard  communications  system  installed  at  all 
facilities. 

Measurement  of  Operating  Parameters 

There  are  two  basic  parameters  to  be  monitored  at 
the  check  sites.  These  parameters  are  water  levels, 
both  upstream  and  downstream  from  the  check  struc- 
ture, and  the  position  of  each  gate  at  the  check. 

Water-Level  Measurement.  Water-level  measure- 
ment is  made  at  float  wells  located  approximately  150 
feet  upstream  and  dow  nstream  from  the  check  struc- 
ture. These  float  wells  have  been  located  to  allow  wa- 
ter-level measurements  to  be  made  at  points  where 
they  are  not  substantially  influenced  bv  water  surface 
fluctuations. 

To  convert  the  water  level  in  the  float  wells  to  a 
quantity  usable  by  the  control  system,  a  perforated, 
stainless-steel  tape  with  a  counterwcighted  float  is 
used  to  drive  a  sprocketed  pulley.  .-K  10-bit,  digital 
encoder  is  attached  to  the  pulley  shaft  to  provide  a 
gray-code  output  of  the  water  level.  Using  this 
method,  water  level  is  measured  to  an  accuracy  of 
plus-or-minus  0.01  foot  over  a  range  of  10  feet.  This 
range  exceeds  the  operating  water-level  fluctuations 
in  the  -Aqueduct. 
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Gate-Position  Measurement.  The  height  of  the 
bottom  of  each  gate  above  the  sill  (the  gate  position) 
is  a  measurement  which,  in  combination  with  water 
levels,  is  used  in  the  calculation  to  determine  flow 
through  the  structure.  To  achieve  the  necessary 
precision  and  accuracy  of  flow  measurement,  it  is 
necessary  to  measure  the  gate  position  to  a  precison  of 
0.1  foot.  Since  many  of  these  gates  are  30  feet  high,  this 
was  a  significant  problem  to  be  solved. 

The  gates  are  moved  by  a  motor-driven  cable  hoist, 
and  it  appeared  possible  to  measure  the  gate  position 
by  attaching  a  digital  encoder  to  measure  the  angular 
position  of  the  cable  sheave.  This  method  proved 
impractical  because  the  differential  stretch  of  the 
cable  between  raising  and  lowering  is  several  times 
greater  than  the  tolerable  error.  Consideration  was 
given  to  driving  a  digital  encoder  with  a  toothed 
pinion  riding  a  rack-mounted  chain  on  the  upstream 
face  of  the  gate,  but  it  was  feared  that  accumulation  of 
debris  in  the  rack  would  render  this  method 
unreliable. 

The  method  finally  devised  uses  a  pantograph  to 
transfer  gate  angular  motion  to  an  instrument  shaft 
which  drives  a  digital  encoder.  The  pantograph, 
shown  in  Figure  14  designated  as  a  "four-bar  linkage", 
has  one  member  (the  gate  arm  link)  rigidly  attached 
to  the  radial  beam  of  the  gate.  The  angular  position  of 
the  gate  arm  link  is  reflected  accurately  in  the  angular 
position  of  another  member   (the  instrument  shaft 


link),  which  is  connected  to  the  check  structure.  The 
instrument  shaft,  driven  by  this  link,  is  coupled  to  an 
encoder  shaft.  Internal  gearing  in  the  encoder 
converts  the  40-  to  60-degree  travel  of  the  gate  to  a  240- 
to  360-degree  travel  of  a  IG-bit  digital  encoder, 
dividing  gate  motion  into  between  680  and  1,024  equal 
angular  parts  with  an  accuracy  of  plus-or-minus  1  bit 
(that  is,  from  one  part  in  680  up  to  one  part  in  1,024, 
depending  on  the  actual  angular  position  of  the  gate) . 

The  output  of  this  encoder  is  converted  into  feet  of 
vertical  gate  opening  (h)  by  the  site  computer  using 
gate  geometry  for  calculation.  The  relationship  is 
nonlinear,  but  an  accuracy  of  at  least  one  part  in  300 
is  maintained  even  in  the  least  accurate  part  of  gate 
motion.  This  results  in  a  measurement  accurate  to  0.1 
foot  for  all  radial  gates  by  using  the  following 
equation. 


h  = 


H  -  R  Sin  0/l_- 
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encoder  output  at 


coder  output  at  h  = 
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where  h   =   vertical  gate  opening  (in  feet) 

H  =  height  of  gate  arm  trunnion  above  the 

invert 
R  =  radius  of  the  radial  gate 
9   =  arc  sin  H/R 
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Figure   14.      Four-Bar    Linkage 
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Local,  Automatic,  Water-Level  Control 

Local,  automatic,  water-level  controllers  have  been 
used  for  many  years  to  control  aqueducts  which 
normally  do  not  have  any  remote  control.  When  used 
with  proper  respect  for  their  limitations,  they  are 
successful  in  maintaining  aqueduct  water-level 
stability  in  the  face  of  disturbances  in  the  steady  flow 
of  water.  They  can  automatically  accommodate 
moderate  changes  in  flow  without  endangering  the 
Aqueduct  due  to  water-level  changes  caused  by  flow 
mismatching.  Such  automatic  adjustment  is  of  real 
value  when  manual  adjustment  requires  a  water 
operator  to  go  to  each  check  structure  to  make  the 
necessary  manual  adjustment. 

The  value  of  local,  automatic,  water-level  control  is 
less  obvious  when  a  remote  control  and  monitoring 
system  is  available.  In  this  case,  an  operator  at  the 
control  center  has  the  opportunity  of  observing  water 
levels,  being  made  aware  of  abnormal  trends  by 
alarms,  and  making  the  required  adjustments  rapidly 
from  a  control  console. 

It  is  only  in  an  emergency,  caused  by  failure  of  the 
remote  control  system  to  one  or  more  of  the  check 
structures,  that  local,  automatic,  water-level  control- 
lers become  useful.  The  original  criteria  for  the  aque- 
duct control  system  assumed  that  such  control  system 
failure  would  occur  and  that  controllers  would  be  able 
to  moderate  the  effects  of  such  failures.  As  a  result  of 
this,  check  structures  No.  1  through  No.  29  are  pro- 
vided with  analog  water-level  controllers,  which  are 
interfaced  to  the  check  structure  control  equipment. 

These  controllers  operate  on  a  proportional  plus 
reset  algorithm, 

d  =  K,e  +  KjT/edt 
where  d  =  change  in  gate  position  at  time  t  =  T 
with  respect  to  gate  position  at  time 
t  =  0 
e  =  water  level  error  (difference  between 
water  level  and  water-level  setpoint  at 
any  time) 
T  =  present  time 
K,  and  Kj  =  constants    based    on    the    physical 
parameters  of  the  individual  sites. 

Along  with  the  automatic  controllers,  certain 
"bypass"  hardware  (which  bypasses  the  computer 
system)  is  provided  to  communicate  water  level  and 
other  key  alarm  information  back  to  the  ACC  in  the 
event  of  a  check  structure  computer  failure. 

Operating  experience  with  early  check  structure 
controls  showed  that  frequent  failures  of  the 
computer  never  materialized.  Failures  do  occur,  but 
they  are  so  infrequent  that  installation  of  the 
controllers  was  considered  to  be  unnecessary.  As  a 
result  of  this  experience,  hardware  local,  automatic, 
water-level  controllers  were  not  included  in 
subsequent  designs  (Check  31  and  beyond). 


To  meet  the  automatic  control  requirements 
deemed  necessary  in  the  Operations  Control  Plan, 
provision  for  addition  of  a  software  water-level  con- 
trol program  has  been  included  in  the  check  structure 
computer  but,  to  date,  its  implementation  has  not 
been  deemed  necessary.  Nevertheless,  all  features  of 
hardware  and  system  operation  are  present  at  all 
check  structures  so  that  local,  automatic,  water-level 
control  can  be  implemented  if  required  at  some  future 
time. 

Since  it  was  believed  that  reliability  would  be  some- 
what impaired  by  depending -on  one  computer  to 
serve  both  as  the  regular  means  of  control  (remote 
control)  and  as  back-up  means  of  control  (local  auto- 
matic control),  the  check  structures  south  of  No.  29 
were  provided  with  a  second  "hot-standby"  computer. 
The  second  computer  was  actually  installed  only  at  a 
few  structures  for  test  purposes  and  has  proven  to  be 
unnecessary  because  of  the  high  reliability  of  the  sys- 
tem with  only  one  computer. 

Subordinate  Site  Interface  Equipment 

In  addition  to  check  structures  and  plants,  there  are 
other  functional  structures  along  the  Aqueduct.  These 
additional  structures  are  chiefly  customer  delivery 
points  (turnouts)  but  also  include  flow  measurement 
stations  and  water  quality  measurement  stations. 
These  structures  generally  have  a  small  number  of 
functions  to  be  controlled  or  monitored  remotely. 

To  have  placed  these  structures  directly  on  the 
control  system  communication  line  as  major  sites 
would  have  resulted  in  extra  costs  for  the  equipment 
required  and  increased  the  complexity  of  the 
communicaton  line.  Consequently,  such  structures 
were  designated  as  "subordinate  sites",  and  provisions 
were  made  to  have  them  interface  with  the  control 
system  through  check  structure  computers.  The  check 
structure  computer  provides  encoding  and  decoding, 
memory,  computation,  and  switching  of  messages 
between  subordinate  sites  and  the  control  center. 

Commands  for  subordinate  sites  received  from  the 
control  center  are  executed  by  the  check  structure 
computer  through  the  subordinate  site  command  in- 
terface. Interrogations  of  subordinate  sites  by  the  con- 
trol center  are  responded  to  with  data  from  the  check 
structure  computer  memory.  Memory  locations  con- 
taining this  data  are  refreshed  periodically  by  a  scan 
of  data  at  the  subordinate  sites  through  subordinate 
site  data  interface. 

The  check-site  computers  are  provided  with 
interfaces  to  communicate  with  subordinate  sites  by 
analog  signals.  The  command  interface  included  a 
digital-to-analog  converter  (DAC)  and  multiplexer 
for  quantitative  commands,  such  as  setpoints. 

The  site  computer,  when  it  receives  a  setpoint 
command  destined  for  a  subordinate  site,  selects  the 
correct  site  on  the  multiplexer  and,  through  the  DAC, 
produces    a    voltage    corresponding    to   a    full-scale 
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"ready"  signal  followed  by  a  voltage  representing  the 
command  setpoint.  These  voltages  are  then  sent  to  the 
subordinate  site  over  telephone  lines  by  use  of 
frequency  modulated  (FM)  tone  transmitters. 
Nonquantized  commands,  such  as  "raise  gate",  are 
output  by  the  check-site  computer  directly  to  a 
specified  relay  and  transmitted  to  the  subordinate  site 
by  a  frequency-shift-keyed  (FSK)  tone  transmitter 
dedicated  to  that  function. 

Data  from  subordinate  sites  are  likewise  received 
via  FM  and  FSK  tone  receivers.  If  quantized,  they  are 
converted  to  a  digital  quantity  by  an  analog-to-digital 
converter  (ADC).  Nonquantized  data  are  passed  on 
by  relay  outputs  of  the  FSK  receivers.  Receiver 
outputs  are  present  at  all  times  and  are  scanned 
periodically  by  the  check-site  computer. 

As  a  result  of  the  inherent  disadvantages  of  analog 
communications,  and  due  to  increasing  availability 
and  decreasing  cost  of  digital  equipment,  the  analog 
interface  is  not  being  used  as  extensively  as  originally 
planned.  Instead,  a  digital  interface  is  used,  with  the 
ADC  and  DAC  located  at  the  subordinate  site. 

Using  this  method,  each  check  structure  acts  as  a 
center  in  a  radial  communication  system  similar  in 
many  respects  to  the  relationship  of  the  check 
structures  and  the  control  center.  Like  the  larger 
system,  the  check  structure  computer  polls  the 
subordinate  sites  with  time-shared  interrogations 
designating  address  and  function  and  receives 
time-shared  responses.  The  method  of  transmitting 
commands,  time  shared  with  the  interrogations,  and 
the  provision  of  security  for  messages  are  also  similar. 

The  most  readily  available  digital  interface,  the 
8-bit,  serial,  asynchronous  interface  of  the  check 
structure  computer,  is  used  to  communicate  with  the 
subordinate  sites. 

Remote  Monitoring  and  Control 

Complete  remote  monitoring  and  control  capability 
has  been  provided  from  the  control  center.  This 
capability  allows  the  operator  in  the  remote  control 
center  to  monitor  and  control  the  check  structure  in 
the  same  manner  as  can  be  done  from  the  operator's 
panel  at  the  site.  Table  5  lists  the  functions  that  are 
exchanged. 

Emergency  Electrical  Power 

Electrical  power  at  the  check  structures  is  obtained 
through  utility  company  electrical  service.  For  back- 
up purposes  when  the  utility  service  fails,  a  liquefied 
petroleum  gas  (LPG)-fueled  engine  generator  is  pro- 
vided at  each  check.  Back-up  power  is  fed  through  a 
transfer-switch  arrangement,  resulting  in  a  short 
power  interruption  whenever  a  transfer  is  made.  To 
enable  satisfactory  operation  after  such  occurrences, 
power-failure  reset  and  automatic  restarting  of  all 
control  equipment  at  the  site  are  included  as  part  of 
the  system. 


Table  5.      Functions  Monitored  and  Controlled  at  a  Check  Structun 


COMMANDS 

Select  Manual  Control 
Select  Automatic  Control 
Telephone  Call  Signal 
Upstream  Water  Level  Setpoint 
Downstream  Water  Level  Setpoint 
Gate  #1  Position  Setpoint 
Gate  §2  Position  Setpoint 
Gate  #3  Position  Setpoint 
Gate  #4  Position  Setpoint 

DATA 

Upstream  Water  Level  Setpoint 

Upstream  Water  Level 

Downstream  Water  Level  Setpoint 

Downstream  Water  Level 

Gate  #1  Position  Setpoint 

Gate  #1  Position 

Gate  #2  Position  Setpoint 

Gate  #2  Position 

Gate  #3  Position  Setpoint 

Gate  #3  Position 

Gate  #4  Position  Setpoint 

Gate  #4  Position 

STATUS 

Automatic  Control  Mode 

Manual  Control  Mode 

Gate  #1  Selected  as  Control  Gate* 

Gate  #2  Selected  as  Control  Gate* 

Gate  #3  Selected  as  Control  Gate* 

Gate  #4  Selected  as  Control  Gate* 

Automatic  Control  from  Upstream  Setpoint 

Automatic  Control  from  Downstream  Setpoint 

Gate  Speed  No.  1  (slow) 

Gate  Speed  No.  2  (medium) 

Gate  Speed  No.  3  (fast) 

Gate  Speed  No.  4  (emergency  close) 

ALARMS 

Loss  of  Communications — Primary 
Loss  of  Communications — Back-up 
Primary  Power  Failure 
Upstream  Water  Level  High 
Downstream  Water  Level  High 
Upstream  Water  Level  Low 
Downstream  Water  Level  Low 
Control  Gate  High  Out-of-Range 
Control  Gate  Low  Out-of-Range 
Unauthoiized  Building  Entry 
Failure  of  Gate  No.  If 
Failure  of  Gate  No.  2\ 
Failure  of  Gate  No.  3t 
Failure  of  Gate  No.  4t 
Computer  Failure 


*  This  status  designates  whicli  Bate  will  be  modulated  by  the  local  a 
troller  to  maintain  the  selected  setpoint.  Selection  is  made  locally. 

t  Failure  of  a  gate  is  indicated  when  the  gate  position  does  not  correspond  t 
gate  position  setpoint  and  the  gate  did  not  move  at  any  time  during  a  two-m 
period. 


Design  of  Pumping  Plant  Control  Systems 

Two  basic  philosophies  exist  in  the  department-de- 
signed pumping  plant  control  systems,  that  is,  com- 
puterized plants  and  noncomputerized  plants. 
Although  the  techniques  used  to  implement  the  con- 
trol systems  are  dissimilar,  the  functional  characteris- 
tics of  each  are  alike. 

The  first  level  of  control  at  all  plants  exists  at  the 
unit  board.  Here,  individual  units  can  be  operated 
independently  without  regard  to  whether  or  not  the 
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plant  control  system  is  in  operation.  Unit  board 
control  is  not  considered  to  be  a  part  of  the  plant 
control  system  (by  definition). 

The  next  level  of  control,  the  plant  control  system, 
consolidates  control  of  all  units,  station  service, 
switchyard,  and  plant  auxiliaries  from  a  central  plant 
control  room.  It  is  the  equipment  in  the  plant  control 
room  that  determines  whether  or  not  a  plant  is 
computerized. 

Normally,  units  and  plant  equipment  are  operated 

from  the  operator's  console  in  the  plant  control  room. 

In  the  following  noncomputerized  plants,  control 

and  monitoring  are  accomplished  through  hardwired 

equipment  and  do  not  make  use  of  a  plant  computer. 

Las  Perillas  Pumping  Plant 

Badger  Hill  Pumping  Plant 

South  Bay  Pumping  Plant 

Del  \'alle  Pumping  Plant 

In  the  following  computerized  plants,  a  control 
computer  performs  the  switching,  translating,  and 
computing  functions  required  to  monitor  and  control 
the  plant. 

Delta  Pumping  Plant 

Buena  Vista  Pumping  Plant 

Wheeler  Ridge  Pumping  Plant 


Wind  Gap  Pumping  Plant 
A.  D.  Edmonston  Pumping  Plant 
Oso  Pumping  Plant 
Pearblossom  Pumping  Plant 

All  plants,  whether  they  are  computerized  or  not, 
have  been  designed  with  capability  for,  or  have  been 
adapted  to,  remote  control  and  monitoring. 

Plants  with  Conventional  Controls 

Department-designed  plants  with  conventional 
(noncomputerized)  controls  are  small  in  capacity 
compared  to  computerized  plants,  the  largest  being 
the  Las  Perillas  and  Badger  Hill  plants  at  450  cubic 
feet  per  second  capacity.  They,  and  the  South  Bay  and 
Del  Valle  Pumping  Plants,  have  been  designed 
without  a  plant  control  room  per  se,  i.e.,  unit  level 
control  only  (Figure  15). 

The  San  Luis  and  Dos  Amigos  plant  control 
systems,  part  of  the  joint-use  facilities,  were  designed 
by  the  U.S.  Bureau  of  Reclamation  (References  5  and 
6)  and  do  not  use  a  plant  computer.  They  are, 
however,  interfaced  with  the  aqueduct  control  system 
for  remote  control  by  a  small  computerized  system 
designed  by  the  Department.  These  two  plants  use 
conventional   controls   with    DC    relays   and   other 


Figure   15.     Typical    Unit    Level    Control    Board 
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electromechanical  devices.  The  central  control 
console  in  the  plant  operates  through  these  devices. 
Data  displays  are  of  the  indicating  or  recording  meter 
type.  An  events  recorder  prints  out  in  numerical  code 
alarms  or  status  upon  occurrence  of  changes,  with 
summaries  occurring  periodically  or  upon  request. 

The  computerized  interface  system  employed  at 
these  two  plants  consists  of  a  small  (8K)  computer 
interfaced  to  the  plant  and  to  the  remote  communica- 
tion system.  The  function  of  the  computer  is  to  scan 
plant  parameters  and  store  data  in  memory  for  trans- 
mission to  the  remote  control  center.  These  plant 
parameters  include  both  digital  and  analog  inputs. 
The  computers  also  respond  to  commands  from  a 
remote  control  center.  These  interface  systems  pro- 
vide the  same  functional  capability  as  the  remote  con- 
trol interconnection  at  a  plant  with  computerized 
controls.  An  exception  to  this  is  that  no  computer 
bypass  capability  exists  at  San  Luis  and  Dos  Amigos 
plants. 

Las  Perillas  and  Badger  Hill  Pumping  Plants,  on 
the  other  hand,  are  provided  with  remote  control 
interconnection  using  hardware  instead  of  a 
computer.  This  hardware  is  an  enhanced  version  of 
the  standard  communications  system  which  has  been 
expanded  to  respond  to  additional  command  and  data 
messages. 

Plants  with  Computerized  Controls 

In  a  computerized  plant,  the  computer  is  the  inter- 
face between  plant  operator  and  plant  equipment  for 
the  purposes  of  data  monitoring  and  local  control.  In 
addition,  the  computer  interfaces  with  the  communi- 
cation system  for  remote  control  and  monitoring.  A 
typical  plant  computer  system  consists  of  from  1 6K  to 
24K  words  of  core  memory,  with  from  32K  to  96K 
words  of  secondary  storage. 

The  functions  of  the  computer  can  be  grouped  into 
categories  of  data  monitoring,  data  display  and 
recording,  computation,  control,  and  remote  control 
interconnection. 

All  six  of  the  computerized  plants  are  similar  in 
function,  although  control  system  hardware  may  be 
different.  The  Delta  Pumping  Plant  was  the  first 
computerized  plant  in  the  Project  and  formed  the 
basis  from  which  other  plant  systems  were  designed. 
It  is  used  as  a  typical  plant  control  system. 

Figure  16  is  a  functional  block  diagram  of  the  Delta 
Pumping  Plant  control  system.  The  computer  used  in 
the  Delta  plant  system  is  a  General  Electric  Model 
4040. 

Data  Monitoring.  Table  6  lists  various  functions 
that  are  monitored  and  controlled  by  the 
computerized  plant  system.  These  are  pump  points, 
motor  points,  valve  points,  transformer  points,  and 
miscellaneous  plant  points.  There  are  three  types  of 
input  signals  from  the  plant  equipment  to  the  plant 
computer — interrupt,  digital,  and  analog.  Interrupt 
inputs  are  immediately  answered  by  the  computer 


and  may  cause  certain  control  routines  to  be 
automatically  executed.  In  addition,  all  points  on 
interrupt  are  functionally  identified  and  logged  in 
their  order  of  occurrence. 

Both  digital  and  analog  inputs  are  scanned  at 
periodic  intervals  by  the  plant  computer.  If  the  point 
is  classified  as  an  alarm,  a  change  in  state  of  the  point 
will  be  annunciated  both  audibly  and  visually.  The 
alarm  also  will  be  printed  out  over  the  alarm  typewrit- 
er with  the  value  of  the  off-normal  condition  included 
for  analog  inputs.  Certain  control  routines  may  be 
initiated. 

If  the  point  is  classified  as  a  status  point,  the  event 
is  printed  out  and  the  change  reflected  on  the  mimic 
bus. 

Data  Display  and  Recording.  Data  are  displayed 
or  recorded  on  logging  typewriters,  digital  displays, 
or  analog  chart  records.  All  of  the  analog  points  as 
listed  in  column  6  of  Table  6  are  logged  periodically. 
Points  which  belong  to  pumping  units  or  major  items 
of  equipment  not  in  use  are  not  included  in  the  logged 
parameters. 

All  alarm  points  and  status  points  listed  in  column 
3  of  Table  6  are  logged  upon  change  in  state.  Analog 
points,  and  those  calculated  values  which  are 
designated  as  alarm  points,  are  compared  to  preset 
limits  and  are  logged  as  alarms  if  an  out-of-limit 
condition  is  detected. 

Certain  special  logging  features  are  available  to  the 
operator  on  demand.  These  consist  of  summary  print- 
outs of  points  currently  in  the  alarm  state  (active 
alarms);  group  review  logs  which  are  arranged  so  the 
operator  may  initiate  a  printout  of  all  analog  inputs 
for  any  major  piece  of  equipment;  or  a  printout  of  all 
analog  inputs  on  a  functional  basis  such  as  tempera- 
tures, pressures,  flows,  and  so  forth. 

The  computer  also  provides  a  log  at  the  completion 
of  unit  start-up  and  prior  to  unit  shutdown  on  demand 
by  the  operator.  An  operator  can  log  any  point  or 
group  of  points  on  demand. 

The  computer  provides  the  capability  to  display  up 
to  six  points  on  digital  displays  or  to  record  (trend) 
up  to  six  points  on  a  series  of  analog  strip-chart 
recorders. 

Computation.  The  computer  provides  the 
computational  capability  necessary  to  scale  all  data 
monitored.  In  addition,  certain  variables,  such  as 
motor  field  temperatures,  are  calculated  from  other 
inputs  available  to  the  computer. 

Control.  The  computer  provides  for  control  of  the 
plant  either  through  the  plant  console  or  remotely  for 
complete  unit  start-up  or  shutdown.  On  a  complete 
computer  start-up,  as  an  example,  the  computer 
performs  all  the  necessary  permissive  checks  prior  to 
start-up,  starts  the  unit  auxiliaries,  depresses  the  water 
level  in  the  pump,  closes  the  unit  breaker,  monitors 
the  unit  starting,  waters  the  pump,  and  opens  the 
discharge  valve. 
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Table  6.      Functions    Monitored    and   Controlled   at 
KEY 


ASA  Device  >'o. 
Description  of  Point 
Instrumentation  Points  (Inputs) 

AP  —  Alarm  Point 

PP  —  Permissive  Point 

SP  —  Status  Point  (Operating  Events) 

DP  —  Data  Point 


Typical   Computerized    Pumping    Plant- 
Column 


Control  Points  (Outputs) 

S  —  Start  or  Stop  Routine  Polr 
W  —  Direct-Wired  Point 
Scan  Period /Interrupt  Point  (INT) 
Periodically  Logged 
Shutdown  Point 

NORM  —  Normal  Shutdown 
EMERG  —  Emergency  Shutdown 


38B 
38H 
38P 
63  R 
63  RH 
63A 
63E 
63G 

63  ZU 
63  ZL 
63  K 
220G 

220H 

220J 

220K 

204E 
204F 
220L 
220M 
220P 
220R 
33T 


12 
26 
99A 
99B 

26F 
26L 

26C 


26T 

38K 

38L 

38J 

38T 

38U 

38V 

46 

48 

49S 

49S 

SIV 

53 

55 

59 

63 

63  P 

64  F 


AP 


PUMP  POINTS 

Pump  Bearing  Temperature 

Pump  Bearing  Hot-Spot  Temperature 

Packing  Box  Temperature 

Pump  Bearing  Low  Oil  Level 

Pump  Bearing  High  Oil  Level 

Packing  Box  Cooling  Water  Pressure. 

Pump  Bearing  Oil  Pressure 

Pump  Bearing  Cooling  Water  Pres- 
sure  

Upper  Seal  Water  Pressure 

Lower  Seal  Water  Pressure 

Pump  Bowl  Water  Level.. 

Open  Pump  Bearing  Cooling  and  Seal 

Water  Valve 

Close  Pump  Bearing  Cooling  and  Seal 

Water  Valve 

Open   Packing   Box   Cooling   Water 

Valve. 

Close   Packing   Box   Cooling   Water 

Valve 

Start  Pump  Bearing  Oil  Pump 

Stop  Pump  Bearing  Oil  Pump 

Open  Water  Depressing  Air  Valves,. 
Close  Water  Depressing  Air  Valves.. 

Open  Air  Release  Valve.. 

Close  Air  Release  Valve 

Air  Release  Valve  Position 


MOTOR  POINTS 

Reverse  Overspeed 

Amortisseur  Temperature 

Field  Current 

Field  Voltage 

Field  Temperature 

Lower  Guide  Bearing  Oil  Tempera- 
ture  

Motor  Cooling  Air  Exhaust  Temper- 
ature  

Motor  Cooling  Air  Exhaust  Temper- 
ature  

Thrust   and   Upper  Guide   Bearings 

Oil  Temperature 

Lower  Guide  Bearing  Temperature. 
Lower  Guide  Bearing  Temperature. 

Thrust  Bearing  Temperature 

Thrust  Bearing  Temperature 

Upper  Guide  Bearing  Temperature. 
Upper  Guide  Bearing  Temperature. 
Negative  Phase  Sequence  Current.. 

Incomplete  Starting  Sequence 

Stator  Temperature 

Stator  Temperature 

Time  Overcurrent 

Field  Failure 

Loss  of  Synchronism 

Neutral  Overvoltage 

Thrust  Bearing  Oil  Level 

Motor  Bearing  Cooling  Water  Pres- 
sure  

Field  Ground 


ISS 

X 

5S 

— 

5S 

— 

5S 

— 

5S 

— 

15S 

X 

15S 

X 

IS 

— 

IS 

— 

— 

X 

— 

X 

15S 

X 

5S 

- 

15S 

- 

5S 

- 

5S 

— 

5S 

— 

15S 

X 

5S 

— 

15S 

X 

15S 

X 

5S 

— 

INT 

— 

INT 

— 

IS 

— 

15S 

X 

INT 

-■ 

INT 

INT 

— 

INT 

— 

5S 

— 

5S 

^ 

INT 

-- 
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Table  6.      Functions   Monitored   and   Controlled   ot   a   Typical   Computerized    Pumping    Plant — continued 


1 

2 

3 

4 

5 

6 

7 

AP 

pp 

SP 

DP 

86M 
87M 

MOTOR  POINTS— CotitinuetJ 

Motor  Lockout  Relay  Position 

X 

X 

X 

X 
X 

X 

X 
X 

X 

X 
X 
X 
X 
X 
X 

X 

X 
X 
X 

X 
X 

X 
X 

S 

s 
s 
s 
s 
s 

s 
s 

INT 
INT 

5S 

158 

SS 

INT 

INT 
INT 
INT 

X 
X 

X 
X 

EMERG 
EMERG 

43  R 

98A 
99C 

Indication  of  Synchronization. _ 

— 

99D 

— 

63C 

NORM 

99E 

— 

99F 

— 

63  M 

CO- Release                              -     

EMERG 

14 



220A 

Release  Motor  Cooling  Water  Dis- 

220B 

close    Motor    Cooling    Water    Dis- 

204A 
204B 

252A 

Start  Thrust  Bearing  Oil  Pump 

Stop  Thrust  Bearing  Oil  Pump 

= 

252B 



63T 

33A 

Thrust  Bearing  High  Oil  Pressure 

— 

33B 
33C 

Motor  Breaker  Assembly  Position... 

— 

33D 

33E 

Exciter  Field  Breaker  Position 

— 

204C 



204D 



63V 

VALVE  POINTS 

X 
X 
X 

X 
X 

X 
X 
X 
X 
X 

X 

X 

X 
X 

X 
X 

- 

s 
s 
s 
s 

s 
s 

SS 
SS 
SS 

- 

63L 



63S 



33F 
33F 
33G 

Spiierical  Valve  Plug  Position  (Open). 
Spherical  Valve  Plug  Position  (Closed) 

33H 



63W 



220U 

Energize    Upstream    Seat    Control 
Valve 

220V 

De-energize  Upstream  Seat  Control 
Valve.   .                          

220W 

Energize  Spherical  Valve  Plug  Con- 

220X 

De-energize    Spherical    Valve    Plug 
Control  Valve 

43B 

Mechanical  Sequencer  Selector  Switch 

204T 

204U 

33V 

Advance  Mechanical  Sequencer 

Reverse  Mechanical  Sequencer 

Mechanical  Sequence  Complete 
(Adv.) 

- 

33W 

Mechanical  Sequence  Complete 
(Rev.) 

33U 

Hydraulic  Auxiliary  Valves  Position.. 

— 

26R 

TRANSFORMER  POINTS 

Transformer  Cooling  Oil  Tempera- 

X 

X 
X 
X 
X 
X 
X 
X 
X 
X 
X 

X 

- 

X 

- 

SS 

SS 
lOS 
INT 
INT 

SS 
INT 
INT 

SS 
INT 
INT 

X 

NORM 

49R 

Transformer  Winding  Hot  Spot  Tem- 

NORM 

49T 
51G 
63D 
63J 
86T 

Transformer  Winding  Temperature.. 

Transformer  Neutral  Ovcrcurrent 

Transformer  Sudden  Pressure 

Transformer  Cooling  Oil  Level 

NORM 
EMERG 
EMERG 

NORM 
EMERG 

87T 
63H 

Transformer  Differential  Current 

EMERG 
NORM 

47 
63EE 

Undervoltage  and  Phase  Sequence 

Transformer  Water  Spray  Released.. 

EMERG 
EMERG 
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Table   6.      Functions    Monitored    and    Controlled    at    u    Typical    Computerized    Pumping    Pic 


SP 


71L 
33J 

33K 
16A 
16B 

64A 
64B 
27A 
50/51 
99G 
99H 
63  N 
63U 
63  F 
63  Y 
26B 
86S 
30HV 
87B 
99J 
99K 
51G 
63X 
204BB 

204V 

204Z 

204W 

204G 

204H 

204K 
204L 
204N 

204P 

224A 

224B 

33? 

33R 

33S 
33Z 

43T 
204J 
204S 

33X 
204X 
204Y 

13B 

33M 

99L 

204AA 

33Y 

27M 

27T 
63AA 


MISCELLANEOUS  POINTS 

Intake  Channel  Water  Level 

Plant  DC  Voltage 

Control  Room  DC  Voltage 

Plant  Battery  Charger  Failure 

Control  Room  Battery  Charger  Fail- 
Plant  Battery  Ground _ 

Control  Room  Battery  Ground 

Oil  Room  CO2  System  Voltage 

Station  Service  Overcurrent 

Station  Service  Kilowatts 

Plant  Kilowatt  Hours 

Plant  Air  Pressure 

Sump  Water  Level 

Water  Flow 

Total  Water  Delivered 

Oil  Storage  Room  Fire 

Station  Service  Lockout 

Heating  and  Air  Conditioning 

Bus  Differential  Current 

Reactive  Volt-Amperes  (Plant) 

Voltage  (Plant)  (230  kV) 

Station  Service  Ground 

Cooling  Water  Pump  Pressure 

Energize  Amplidyne  Transfer  Con- 
tactor  

De-energize      Amplidyne      Transfer 

Contactor __ - 

Energize  Amplidyne  PT  and  CT  Con- 
tactor  

De-energize  Amplidyne  PT  and  CT 

Contactor 

Start  Cooling  Water  Pump  and 

Strainer 

Stop  Cooling  Water  Pump  and 

Strainer 

Close  Switchyard  Breaker 

Open  Switch}"ard  Breaker 

Close  Station  Service  13  .8  kV  Break- 


Open  Station  Service  13  .8  kV  Break- 


Close    Motor-Operated    Disconnect 
Switch 

Open     Motor-Operated     Disconnect 

Switch 

Switchyard  Breaker  Position 

Station  Service  13.8  kV  Breaker  Po- 
sition   -- 

Motor-Operated   Disconnect   Switch 

Position  (Open) 

Motor-Operated   Disconnect   Switch 

Position  (Close) 

Control  Transfer  Switch  Position 

Start  375  CFM  Air  Compressors 

Stop  375  CFM  Air  Compressors 

Radial  Gate  Position 

Start  Amplidyne  M-G  Set 

Stop  Amplidyne  M-G  Set — 

M-G  Set  Speed 

Amplidyne  PT  and   CT  Contactor 

Position 

Amplidyne  Voltage 

Operate  Amplidyne  Voltage  Adjuster 
Amplidyne  Transfer  Contactor  Posi- 
tion   

Unit  DC  Voltage _. 

375  CFM  Air  Compressor  Status.  .- 

Bay  DC  Voltage _-- 

Air  Compressor  Cooling  Water  Pres- 


L\T 

3S 
5S 
5S 

5S 
5S 
5S 
5S 
L\T 


INT 

INT 

5S 

INT 


5S 
15S 


INT 

INT 

INT 

INT 
INT 


NORM 
EMERG 


NORM 


EMERG 
EMERG 
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The  operator  can  also  request  a  partial  unit  start  to 
verify  proper  unit  permissives  and  unit  auxiliary 
operation  prior  to  the  need  to  perform  a  complete 
start-up. 

Normal  shutdown  of  a  unit  can  be  accomplished 
either  bv  the  operator  through  the  plant  console,  from 
a  remote  stop  command,  as  a  result  of  internal  action 
on  an  alarm  point  which  is  not  aborted  by  the 
operator,  or  as  a  result  of  an  aborted  unit  start-up 
which  requires  a  return  to  the  shutdown  state. 

When  an  emergency  shutdown  occurs,  it  usually 
results  from  the  unit's  protective  relaying  or  by  direct 
trip  either  by  the  operator  or  remotely.  The  motor 
breaker  is  tripped  and  the  main  discharge  valve  is 
automatically  closed  by  equipment  external  to  the 
control  system.  The  computer  automatically  provides 
back-up  to  this  external  equipment  to  shut  down  the 
auxiliary  equipment  of  the  tripped  unit(s). 

Direct-wire  control  (bypassing  the  computer) 
exists  for  certain  critical  functions  at  the  plant.  These 
controls  are  as  follows: 


1.  Open  and  close  each  motor-operated  disconnect. 

2.  Open  and  close  each  switchyard  circuit  breaker. 

3.  Open  and  close  each  station  service  high-voltage 
breaker  and  maintenance  yard  feeder  high-voltage 
breaker. 

4.  Reset  and  silence  all  annunciators  and  alarms 
located  in  the  control  room  and  plant. 

5.  Emergency  trip  each  motor  circuit  breaker. 

Operator  communication  with  the  control  svstem  is 
through  the  operator's  console  shown  in  Figure  17. 
This  console  provides  a  mimic  panel  (Figure  18)  de- 
picting the  plant  single-line  diagram,  with  direct-wire 
controls  installed.  Fhe  console  also  contains  various 
thumb  uheel  switches  and  pushbuttons  (Figure  19) 
through  uhich  the  operator  communicates  uith  the 
computer.  Finally,  the  logging  typewriters,  digital  dis- 
plays, and  analog  recorders  are  contained  in  the  con- 
sole. Figure  20  shows  the  digital  displays  and  analog 
recorders. 


Figure   17.      Operator's   Console — Delta    Pumping    Plant   Control   Systen 
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Figure   18.     Mimic    Panel — Delta    Pumping    Plant    Control    System 
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Figure    19.     Operator    Communication— Delta    Pumping    Plant    Control    Syste 
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Figure  20.      Digitol    Displays    and    Analog    Recorders — Delta    Pumping    Plant    Control    Systen 


Remote  Control  Interconnection.  Control  and 
monitoring  of  plants  at  the  plant  control  room 
provides  for  a  larger  number  of  points  to  be  available 
at  the  plant  control  console  and  for  frequent  updating 
of  monitored  variables.  When  control  is  exercised 
from  a  remote  location,  such  as  the  ACC  or  the  POCC, 
practical  considerations  require  a  reduction  in  the 
number  of  points  monitored,  in  the  frequency  of 
updating,  and  in  the  variety  of  controls  available.  To 
transmit  plant  data  back  to  the  ACC  would  result  in 
a  communications  requirement  significantly  greater 
than  the  capacity  of  the  lines  used  on  the  aqueduct 
system. 

Pumping  plants  can  be  operated  remotely  with 
much  less  data  than  are  required  at  the  plant  itself. 
Table  7  shows  the  command,  data,  and  status 
information  exchanged  between  the  remote  control 
center  and  the  plant  system. 

Alarms  transmitted  from  the  plant  to  the  remote 
control  center  convey  less  detailed  information  than  is 
available  at  the  plant  control  room.  The  concept  of 
remote  control  of  a  plant  is  one  of  dispatch  rather  than 
supervisory  control.  The  primary  concern  of  the 
operator  in  the  remote  control  center  is  whether  or 
not  a  unit  is  available  for  service.  Since  the  units  and 
other  features  of  the  plant  are  protected  from  damage 
by  local   protective  devices,  the  quantity  of  alarm 


information  transmitted  remotely  is  held  to  a 
minimum.  Plant  and  unit  alarms  are  grouped  by 
function,  and  it  is  the  general  alarm  which  is 
transmitted  remotely.  Table  8  lists  the  alarms  which 
are  transmitted  to  the  ACC. 


Table  7.      Information  Exchanged  Between  an  ACC  and  o 
Typical  Pumping  Plant  System 

COMMANDS 

Start  Unit — Each  unit 

Stop  Unit  (normal) — Each  unit 

Stop  Unit  (emergency) — Each  unit 

Open  Breaker — Each  switchyard  breaker 

Close  Breaker — Each  switchyard  breaker 

DATA 

Intake  Water  Level 
Penstock  Flow  (cfs) — Each  penstock 
Line  Voltage — Each  transmission  line 
Megawatts — Each  transmission  line 
Vars — Each  transmission  line 
Unit  Megawatts — Each  unit 
Unit  Vars — Each  unit 
Plant  Power  Demand 

STATUS 

Status  of  the  Headworks  Gates;  open,  closed 
Status  of  the  Switchyard  Breakers;  open,  closed 
Unit  I/S  (In-Service) 
Unit  S/D  (Shutdown) 
Unit  0/S  (Out-of-Service) 
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Remote  communication  is  handled  by  a  standard 
communication  system  which  is  interfaced  to  the 
plant  computer.  In  addition,  this  communication 
equipment  provides  a. computer  bypass  capability  for 
remote  emergency  shutdown  of  the  pumping  units. 
To  accomplish  this  bypass  function,  communication 


terminal  equipment  has  been  enhanced  to  provide  site 
code  recognition  and  checking  of  the  message  security 
code  (parity).  It  also  "gates"  command  signals  to  the 
proper  plant  control  contacts.  This  bypass  feature  is 
provided  to  allow  emergency  shutdown  of  the  units 
when  the  plant  computer  is  out  of  service. 


Toble  8.      Alarms   Transmitted    Between   a    Typical    Pumping    Plant   System   and   the   ACC 


ALARM  WHICH  IS  TRANSMIHED  REMOTELY 


PLANT  CONDITION  WHICH  CAUSES 
ALARM  TO  BE  SENT 


Plant  Alarms 

Plant  Local  Control 

Primary  Power  Fail _ 

Plant  Warning.- 


Bus  Differential 

Switchyard  Breaker... 
Trip  (Each  Breaker) 

PG&E  Undervoltage.. 
Transformer  Lockout. 

(Each  Transformer) 
Transformer  Trouble.. 

(Each  Transformer) 


Unit  Alarms  (Each  Unit) 
Pump  Temperature 

Un  i t  Lockout 

Motor  Temperature 


Reverse  Overspeed. 
Electrical  Fault 


COj  Discharge 
Start  Failure.. 
Unit  Warning. 


ACC  Lockout  Set 

PG&E  Load  Shed  Relay 

Station  Service  Lockout 

Oil  Storage  Room  Fire 

Forebay  Water  Level  High 

Plant  Air  Pressure 

Sump  Water  Level  High 

Bay  DC  Voltage 

Air  Compressor  Cooling  Water  Pressure 

Switchyard  DC  Failure 

230  kV  Bus  Differential 

Line  Distance  Relay 

Carrier  Distance  Ground 

Breaker  SFe  High  Pressure 

Overcurrent 

Carrier  Operate 

PG&E  Undervoltage 

Transformer  Lockout 

Oil  Temperature 
Hot  Spot 
Hot  Spot  Trip 
Winding  Temperature 
Neutral  Current 
Fault  Pressure 
Water  Spray 
Pressure  Relief 
Differential  Current 
Oil  Level 
Undervoltage 


Pump  Bearing  Temperature 

Intermediate  Bearing  Temperature 

Packing  Box  Temperature 

Motor  Breaker  Tripped  &  Lucked  Out 

Lower  Guide  Bearing  Oil  Temperature 

Lov.'cr  Guide  Bearing  Temperature 

Upper  Guide  Bearing  Temperature 

Thrust  Bearing  Temperature 

Thrust  &  Upper  Guide  Bearing  Oil  Tcmperalur 

Motor  Cooling  Air  Exhaust  Temperature 

Motor  Bearing  Cooling  Water  Pressure 

Stator  Temperature 

Air  Cooler  Water  Pressure 

Reverse  Overspeed 

Negative  Phase  Sequence 

Time  Overcurrent 

Field  Failure 

Loss  of  Synchronism 

Neutral  Overvoltage 

Field  Ground 

Differential  Current 

C02  Discharge 

Incomplete  Sequence 

Unit  DC  Voltage 

Discharge  Valve  Downstream  Scat  Position 

Thrust  Bearing  Oil  Level 

Pump  Bearing  Oil  Level 

Hydraulic  System  Pressure 

Accumulator  Oil  Level 
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Design  of  Power  Plant  Control  Systems 

Two  power  plant  control  systems  have  been 
designed  by  the  Department,  the  Oroville-Thermalito 
system  and  the  Devil  Canyon  system. 

Oroville-Thermalito  Control  System 

The  Oroville-Thermalito  control  system  provides 
remote  supervisory  control,  automatic  load-control, 
and  data  acquisition  for  Edward  Hyatt  and 
Thermalito  Powerplants,  Oroville  Switchyard,  11 
hydraulic  structures,  and  12  gauging  stations. 

The  Oroville-Thermalito  control  system  includes 
equipment  in  Edward  Hyatt  Powerplant,  Thermalito 
Powerplant,  and  associated  hydraulic  structures,  as 
well  as  necessary  connecting  lines.  Figure  21  shows 
the  location  of  the  Oroville-Thermalito  facilities. 

Edward  Hyatt  Powerplant  is  located  underground, 
beneath  the  left  abutment  of  Oroville  Dam.  The 
Oroville  Control  Building  is  located  aboveground, 
next  to  Oroville  Switchyard  and  approximately  2,000 
feet  from  Edward  Hyatt  Powerplant. 

All  features  of  Thermalito  Powerplant  are  located 


aboveground,  approximately  8  miles  from  the 
Oroville  ACC. 

The  Oroville-Thermalito  control  system  is 
normally  operated  from  the  Oroville  ACC  (Figure 
22)  but,  in  the  event  of  an  emergency,  operation  is 
possible  from  an  emergency  control  room  located  in 
the  underground  Edward  Hyatt  Powerplant. 

The  Oroville-Thermalito  control  system  is 
composed  of  three  supervisory  subsystems,  three 
telemetry  subsystems,  a  data  logging  subsystem,  an 
alarm  events  and  status  change  subsystem,  and  an 
automatic  load-control  subsystem.  Figure  23  is  a  block 
diagram  of  the  Oroville-Thermalito  control  system. 

Supervisory  Subsystems.  Three  supervisory  sub- 
systems are  provided  for  remote  control  of,  and  select- 
ed data  acquisition  from,  unattended  facilities  of  the 
Oroville-Thermalito  power  complex. 

One  subsystem  is  assigned  to  Edward  Hyatt  Power- 
plant  and  one  gauging  station.  This  subsystem  is  ac- 
tivated from  its  master  station  in  the  Oroville  ACC. 
Commands  are  executed  and  data  collected  by  the 
remote  station  located  in  Edward  Hyatt  Powerplant. 
Figure  24  shows  the  master  station  console. 


Figure  22.     Oroville    Area    Control    Center 
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Figure  24.     Master    Console — Edward    Hyatt    Powerpfont    Supervisory    Subsystem 


The  second  subsystem  is  assigned  to  Thermalito  Edward  Hyatt  Powerplant.  Commands  for  the  second 

Powerplant,  six  hydraulic  structures,  and  five  gauging  subsystem  are  executed  and  data  collected  by  a  remote 

stations.  This  subsystem  can  be  activated  from  either  station  located  at  Thermalito  Powerplant  and  remote 

of  two  master  stations,  one  of  which  is  in  the  Oroville  stations  at  the  associated  hydraulic  structures.  Figure 

ACC  and  the  other  in  the  emergency  control  room  in  25  shows  the  master  station  at  the  ACC. 
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Figure  25.     Master    Console — Thermalito    Powerplont    Supervisory    Subsysten 
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The  third  subsystem  is  assigned  to  Oroville 
Switchyard,  five  hydraulic  structures,  and  six  gauging 
stations.  This  subsystem  is  activated  from  its  master 
station  in  the  emergency  control  room  of  Edward 
Hyatt  Powerplant.  Figure  26  shows  this  master 
console.  Associated  hydraulic  structures  can  also  be 
activated  from  the  hydraulic  structures  master  station 
(Figure  27)  located  in  the  Oroville  ACC.  Commands 
for  the  third  subsystem  are  executed  and  data 
collected  by  the  remote  station  located  in  the  Oroville 
ACC  and  remote  stations  at  associated  hydraulic 
structures. 

Hydraulic  structures  and  gauging  stations  of  the 
Oroville-Thermalito  power  complex,  which  are 
controlled  and  monitored  through  three  supervisory 
subsystems,  are  located  in  the  vicinity  of  Oroville 
Dam,  on  the  Feather  River  between  Oroville  Dam  and 
Thermalito  Powerplant,  and  around  Thermalito 
Forebay  and  Afterbay. 

The  Oroville  Dam  core  block  gauging  station  is 
controlled  through  the  Edward  Hyatt  Powerplant 
supervisory  subsystem.  Features  near  Oroville  Dam 
and  on  the  Feather  River  are  controlled  through  the 


Figure   26.      Moiter    Console — Oroville    Switchyard    Supervisory    Subsysten 


Figure  27.      Hydraulic    Structure    Master    Console 
Oroville    Switchyard    Supervisory    Subsystem 


Oroville  Switchyard  supervisory  subsystem.  Features 
around  Thermalito  Forebay  and  Afterbay  are 
controlled  through  the  Thermalito  Powerplant 
supervisory  subsystem.  The  telemetered  variables  are 
hydraulic  in  nature  and  are  displayed  in  digital  form 
at  appropriate  master  stations. 

There  are  four  small  computers  (all  are  Digital 
Equipment  Corporation  Model  DPD-8/S)  in  the 
three  supervisory  subsystems,  one  computer  being  in- 
stalled in  each  of  the  four  master  stations.  One  com- 
puter is  installed  in  the  Edward  Hyatt  Powerplant 
master  console  and  is  assigned  to  that  subsystem.  The 
second  computer  is  installed  in  the  Oroville  Switch- 
yard master  console  and  is  assigned  to  that  subsystem. 
The  third  and  fourth  computers,  assigned  to  the  Ther- 
malito Powerplant  subsystem,  are  installed  in  two 
separate  master  consoles.  One,  installed  in  the  Ther- 
malito Powerplant  master  console,  is  located  in  the 
emergency  control  room.  The  other,  installed  in  the 
hydraulic  structures  master  console  in  the  Oroville 
A("C,  is  assigned  not  only  to  the  Thermalito  Power- 
plant  subsystem  but  also  to  hydraulic  structures  of  the 
Oroville  Switchyard  subsystem. 
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The  following  tables  summarize  the  control  and 
monitoring  capability  through  the  Edward  Hyatt 
Powerplant,  Thermaiito  Powerplant,  and  Oroville 
Switchyard  supervisory  subsystems. 

The  points  in  each  subsystem  that  pertain  to  a  sin- 
gle pumping  unit  or  other  piece  of  apparatus  are 
designated  as  "unitized".  The  points  in  each  subsys- 


tem that  are  related  to  ail  units  or  apparatus  are  desig- 
nated as  "nonunitized". 

Table  9  shows  capabilities  through  the  Edward 
Hyatt  Powerplant  supervisory  subsystem.  Table  10 
shows  capabilities  through  the  Thermaiito  Power- 
plant  supervisory  subsystem.  The  capabilities 
through  the  Oroville  Switchyard  subsystem  are 
shown  in  Table  11. 


Table  9.     Supervisory    Capability — Edward    Hyatt    Powerplant 


IDENTIFICATION 


COMMAND  AND  STATUS 


UNIT- 
IZED 


NON- 
UNIT- 
IZED 


CI 
CI 
C 
C 
CI 
C 

c 

CI 
CI 

I 
I 
I 

CI 

CI 
CI 
CI 

I 
I 
I 
I 
I 

CI 
CI 
CI 
CI 

I 
I 
I 

CI 
CI 
CI 
CI 
CI 

I 
I 


Master  Control _. 

Condenser  Programming* 

Load  Limit  (Wicket  Gate  Limit) 

Speed  Level 

Bypass  Dashpot 

Field  Rheostat 

Voltage  Adjuster.  _ 

Auto  Synchronizert 

Emergency  Unit  Shutdown 

Partial  Shutdown 

Shutdown 

Transfer  to  Local  Control 

Penstock  Valve 

15  kV  Disconnect  Switch  GeneratorJ.. 

Load  Control 

Auto  Synch  ionizer 

15  kV  Disconnect  Switch  Pump§ 

Mode  Orientation  _ 

Mode  Orientation 

Master  Station  Control 

Xfer  to  Emergency  Generator 

Core  Block  Sump  Pump  No.  !__ 

Core  Block  Sump  Pump  No.  2 

Core  Block  Sump  Pump  No.  3 _ 

Core  Block  Water  Level 

Master  Check 

Breaker  UIB 

Powerplant  Gate  Lock 

Station  Service  P3A1 

Station  Service  P3A2 

Station  Service  P4A1 _. 

Station  Service  P4A2 

Station  Service  Pwr  Xfer 

Spherical  and  Dispersion  Valves  No.  1 
Spherical  and  Dispersion  Valves  No.  2 


Start/Stop/Check — Start  Sequence — Complete/Unit. 

ON/OFF— Cond/Gen/Reduced  Freq 

Raise/Lower/Check 

Raise/Lower/Check 

Open/Close — Close/Open 

Raise/Lower/Check 

Raise/Lower/Check 

ON/OFF— Main  Breaker 

Trip  and  Annunication _- 

Unit  Tripped  to  Speed  No  Load 

Incomplete  Sequence/Unit  Creeps 

Local  Control ._ — 

Open/Close — Close/Open 

Open/Close — Close/Open 

Local  Manual/Auto  Control 

ON/OFF— Main  Breaker 

Close/Open 

Blocking  Gen — Pwr  Complex/Gen _ 

Blocking  Pumping — Pwr  Complex/Pumping_._ 

Master  Station — Xfer  Control 

Xfer  to  Emergency  Generator 

Pump  Running — Start/Stop 

Pump  Running — Start/Stop 

Pump  Running — Start/Stop 

High  Water  Level — Water  Level _- 

Master  TLM,  Remote  TLM,  and  Command  Errors-. 

Close/Trip 

Close/Trip -- 

Close/Trip — 

Close/Trip -- 

Close/Trip  - — 

Close/Trip - 

Close/Trip 

Close/Open - - 

Close/Open - 


•  Cond/Gen/Pump— ON/OFF— for  Units  2,  4.  and  6  (CI), 
t  Transfer  Breaker— ON'/OFF— for  Units  2.  4,  and  6(CI). 
J  Indication  ONLY— Units  2,  3,  5,  and  6. 
I  Indication  on  Units  2  and  6— CI  on  Unit  4. 


CONTROL 

INDICATION 

CONTROL  and  INDICATION 
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Table   10.     Supervisory   Capability — Thermalito   Powerplant 


IDENTIFICATION 


COMMAND  AND  STATUS 


UNIT- 
IZED 


Master  Control 

Condenser  Programming* .- 

Load  Limit  (Wicket  Gate  Limit)  . 

Speed  Level 

Bypass  Dashpot 

Load  Control 

Field  Rheostat 

Voltage  Adjuster -- 

Emergency  Shutdown 

Unit  Tripped  to  Speed  No  Load.. 

Incomplete  Sequence 

Transfer  to  Local  Control 

13.8  kV  Breaker  (Gen) 

Kaplan  CAM  Selector.  . 

Governor  Transfer  Valve  Position 

Transfer  Fault 

230  kV  Load  Interrupter  Switch.. 

Creep  Detector 

13.8  kV  Pumping  Breakerf 

Mode  Orientacion 

Mode  Orientation 

Master  Station  Transfer  Control. 
Master  Check 

Power  Plant  Gate  Lock 

Station  Service  Power  Transfer.. 

Western  Canal 

Richvale  Canal _ 

PG&E  Lateral  Outlet.. 

Sutter  Butte  Canal 

Feather  River  Outlet 

Thermalito  Tailrace  Water  Level. 

Thermalito  Bypass  Gate 

Thermalito  Forebay  Water  Level. 

230  kV  Breaker  DWl  (562) 

230  kV  Breaker  BW2  (662) 


Start/Stop/Check — Start  Sequence — Stop  Sequence. 

ON/OFF— Gen/Cond 

Raise/Lower/Check 

Raise/Lower/Check 

Close/Open. _ 

Local/Manual — Auto  Control 

Raise/Lower/Check 

Raise/Lower/Check 

Trip. 


Tripped  to  Speed  No  Load 

Start/Stop 

Transfer  to  Local  Control 

Close/Trip 

Raise/Lower — Analog  Position. 

Main/Aux 

Fault 


Close/Open 

Creep 

Close/Trip — 

Blocking  Gen — Pwr  Complex  Gen 

Blocking  Pump — Pwr  Complex  Pump 

Control  Transfer 

Master     Remote — Hydraulic    Tel/Alarm — Command 
Error 

Gate  Lock 

Pwr  Transfer 

Send/Check — Auto/Manual ^^ 

Send/Check— Auto/Manual 

Send/Check — Auto/Manual 

Send/Check— Auto/.Manual 

Send/Check — Auto/Manual 

Check — Level 

Check — Close/Open 

Level 

Close/Trip 

Close/Trip 


•  Cond/Gen/Pump— ON/OFF— for  Unll 
t  Indication  ONLY  Units  2,  3,  and  4. 


C   =  CONTROL 
I   =  INDICATION 
CI   =  CONTROL  and  INDICATION 


Table   11.     Superv 

sory   Capability — Oroville   Switchyard 

TYPE 

IDENTIFICATION 

COMMAND  AND  STATUS 

UNIT- 
IZED 

NON- 
UNIT- 
IZED 

I 

Total  Megawatts  and  Megavars  for  Oroville  and 
Thermalito  Display  MW  and  MVARS  for  each 
plant,. 

- 

X 

I 

Oroville  Dam — Seepage  Measurement  Weir 

Fish  Barrier  Water  Level 

Check— Level- 

X 

I 

X 

I 

Tailrace  Water  Level  Oroville 

Level/Hi/Check— Water/Alarm.   . 

X 

CI 

Oroville  Flood  Control  Outlet 

X 

CI 

Thermalito  Diversion  Dam  Spillway 

X 

CI 

Thermalito  Power  Canal 

X 

CI 

Palermo  Outlet . 

X 

I 

Master  Check 

Master/Remote— Hydraulic/TLM/Alarm— Command 

Trashracks  No.  1  and  No.  2 

X 

I 

X 
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Telemetering  Subsystems.  Three  independent 
telemetering  subsystems  are  provided  for  transmit- 
ting data  and  command  signals  over  the  communica- 
tion channels  between  various  facilities.  These 
subsystems  also  convert  the  signals  into  suitable  form 
for  interfacing  with  transmitting  and  receiving  equip- 
ment. 

Each  subsystem  is  capable  of  transmitting  and  re- 
ceiving both  analog  and  digital  signals.  The  analog 
telemetering  is  of  the  variable-frequency  type,  with 
the  frequency  rate  change  not  exceeding  35  hertz.  The 
digital  telemetering  is  a  three-tone,  frequency-shift- 
keying  type,  with  a  channel  band  width  of  170  hertz. 

Each  subsystem  includes  tone  receivers  and 
transmitters,  power  supplies,  tone  generators, 
scanning  and  multiplexing  devices,  converters,  line 
drivers,  termination,  and  protection.  Each  subsystem 
also  has  components  necessary  for  interfacing  with 
equipment  of  the  supervisory,  automatic  load-control, 
data  logging,  and  alarm  events  and  status  change 
sequence  subsystems. 

The  first  telemetering  subsystem  is  used  in 
conjunction  with  the  supervisory  subsystem  for 
digital  transmission  of  commands,  alarms,  status 
changes,  and  requests  for  information  between 
associated  master  and  remote  stations. 

The  second  telemetering  subsystem  is  used  in 
conjunction  with  the  data  logging  subsystem  and 
recorders  mounted  on  the  graphic  board  located  in  the 
ACC.  This  subsystem  continuously  scans  the 
metering  instruments  at  both  powerplants,  Oroville 
Switchyard,  Oroville  intake  structure,  and  Oroville 
flood  control  outlet. 

The  third  telemetering  subsystem  is  used  in 
conjunction  with  the  alarm  events  and  status  change 
sequence  subsystem  and  the  automatic  load-control 
subsystem.  This  continuously  scans  three  remote 
memories  and  assigned  recorders. 

Data  Logging  Subsystem.  The  purpose  of  the  data 
logging  subsystem  is  to  record  operational  data  ac- 
quired from  specified  facilities  of  the  Oroville-Ther- 
malito  power  complex.  Data  acquired  by  the  data 
logging  subsystem  are  collected,  processed,  and  fed  to 
a  teletypewriter  and  strip-chart  recorders. 

The  data  logging  subsystem  receives  digital  data 
from  Edward  Hyatt  Powerplant,  Thermalito  Power- 
plant,  Oroville  Dam  flood  control  outlet,  and  Edward 
Hyatt  Powerplant  intake  structure.  The  subsystem 
receives  analog  signals  from  Oroville  Switchyard. 

Digital  data  are  transmitted  to  the  master  station  in 
the  Oroville  ACC  via  a  three-tone  FSK  telemetry  link 
at  a  bit  rate  of  35  bits  per  second.  Data  from  Edward 
Hyatt  Powerplant  contain  unit  megawatts,  unit 
megavars,  unit  water  flow  (in  and  out),  and  water 
level  in  Lake  Oroville. 

Unitized  data  from  Thermalito  Powerplant  contain 
megawatts,  megavars,  megawatt-hours  (in  and  out), 


water  flow  (in  and  out),  wicket-gate  position,  Kaplan 
cam  identification  (Unit  1  only),  highest  bearing 
temperature,  and  highest  stator  temperature.  Data  are 
also  monitored  on  total  power  plant  megawatts,  total 
megavars,  kilovolts,  frequency,  total  water  flow  (in 
and  out),  and  water  levels  in  the  Thermalito  tailrace 
and  Thermalito  Forebay. 

Data  from  the  Oroville  Dam  flood  control  outlet 
contain  information  on  the  position  of  eight  gates. 
Data  from  the  Edward  Hyatt  Powerplant  intake 
structure  contain  temperature  information  from  24 
thermocouple  devices.  Data  from  Edward  Hyatt 
Powerplant  tailrace  and  Oroville  Dam  seepage  weir 
contain  information  on  the  water  levels  at  these  sites. 

Analog  data  from  Oroville  Switchyard  contain 
information  on  megawatts  and  megavars  from  each  of 
three  230-kV  transmission  lines,  total  megawatts  and 
megavars,  system  voltage  and  frequency,  temperature 
from  1 2  points  including  each  of  the  six  2  30-k V  cables, 
and  two  alarms  (one  for  excessive  cable  temperature 
and  one  for  cable  system  air  temperature). 

Data  received  are  processed  by  a  small  computer 
(an  8K  word,  Varian  Model  6201)  and  are  fed  to  the 
logging  teletypewriter  and  back-up,  strip-chart 
recorders. 

Alarm  Events  and  Status  Change  Sequence 
Subsystem.  The  purpose  of  the  alarm  events  and 
status  change  sequence  subsystem  is  to  record,  in  time 
ordered  sequence,  system  operations  during  fault 
conditions.  This  subsystem  and  the  data  logging 
subsystem  use  the  same  computer,  located  in  the 
master  station  in  the  Oroville  ACC. 

The  alarm  events  and  status  change  sequence 
subsystem  records  selected  alarms  and  status  changes 
from  Edward  Hyatt  Powerplant,  Thermalito 
Powerplant,  and  Oroville  Switchyard.  Alarms  and 
changes  in  status  for  unitized  and  nonunitized 
contacts  are  sensed,  combined  with  time  of  change, 
and  transmitted  to  the  data  logging  equipment  in  the 
Oroville  ACC  for  printout.  Alarms  and  events 
monitored  are  shown  in  Tables  12,  13,  and  14. 

The  master  station  for  the  subsystem,  located  in  the 
Oroville  ACC,  controls  and  time-synchronizes  each  of 
the  three  remote  scanning  stations. 

Automatic  Load-Control  Subsystem.  The  auto- 
matic load-control  subsystem  is  composed  of  two  sepa- 
rate parts:  (1)  a  load-control  console  located  in  the 
Oroville  ACC,  and  (2)  ten  unit  controllers  located  in 
the  emergency  control  room  in  Edward  Hyatt 
Powerplant  and  in  the  duplex  switchboard  at  Ther- 
malito Powerplant.  Power  generated  by  the  Oroville- 
Thermalito  complex  can  be  controlled  by  the  auto- 
matic dispatch  system  (ADS)  of  Pacific  Gas  and  Elec- 
tric Company  located  in  San  Francisco.  The  ADS 
originates  a  load  signal  which  is  transmitted  to  the 
Oroville-Thermalito  load-control  subsystem  using  mi- 
crowave and  powerline  carrier  equipment.  This  sig- 
nal is  received  by  the  load-control  console.  It  is  the 
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Table   12.     Alarms  and  Events  Monitored — Edward  Hyatt  Powerplont 


Gov.  Tank  lx)w  Oil  Level 

Gov.  Low  Oil  Pressure 

Gov.  Sump  Tank  Low  Oil  Level 

Gov.  Standby  Oil  Pump  Started 

Turbine  DC  Oil  Pump  Operating 

Turbine  Guide  Bearing  High  Metal  Temp 

Turbine  Guide  Bearing  Temp  Relay 

Turbine  Guide  Brg.  Cooling  Water  Low  Flow 

Turbine  Guide  Brg.  Discharge  Oil  Temp 

Turbine  Guide  Brg.  High  Oil  Level 

Turbine  Guide  Brg.  Low  Oil  Level 

Turbine  Packing  Box  Temp 

Gen.  Upper  Guide  Brg.  Metal  High  Temp 

Gen.  Lower  Guide  Brg.  Metal  High  Temp 

Gen.  Lower  Guide  Brg.  Water  Low  Pressure 

Gen.  Lower  Guide  Brg.  Oil  High  Temp 

Gen.  Lower  Guide  Brg.  Water  Low  Flow 

Gen.  Thrust  Brg.  Metal  High  Temp 

Gen.  Thrust  Brg.  High  Temp 

Unit  Brg.  High  Temp 

Gen.  Thrust  Brg.  Water  Low  Pressure 

Gen.  Thrust  Brg.  Low  Oil  Pressure 

Gen.  Thrust  Brg.  High  or  Low  Oil  Level 

Gen.  Thrust  Brg.  Oil  High  Temp 

Gen.  Thrust  Brg.  Water  Low  Flow 

TSV  Comp.  Cooling  Water  Low  Flow 

Runner  Seal  Ring  Cooling  Water  Low  Flow 

TSV  Comp.  Extreme  High  Pressure 

TSV  Press.  Tank  Low  Oil  Level 

TSV  Water  Filter  Differential 

TSV  Sump  Low  Oil 

TSV  Low  Air  Pressure 

TSV  Tap  Guard  Valve  ."Mr  Revr.  Low  Press 

TSV  Tap  Guard  Valve  Close 

Main  Transformer  Sudden  Pressure 

Gov.  Air  Pressure  Failure 

Unit  Cooling  Water  Strainert 

230  kV  Cable  Pilot  Wire  Monitoring  Relay 


UNITIZED  POINTS,  TYPICAL  EACH  UNIT 

Volts  Per  Cycle  (T) 

Main  Transformer  Winding  Temp  RTD 

Main  Transformer  Winding  Temp 

Main  Transformer  Oil  High  Temp 

Transformer  Differential  &  CO;  Release 

Main  Transformer  Differential 

Main  Transformer  Low  Oil  Flow 

Main  Transformer  Cooling  Water  Low  Flow 

Main  Transformer  Low  Oil  Level 

Main  Transformer  Press  Relief  Device 

Main  Transformer  Tank  Low  Press 

Main  Transformer  Cvlinder  Low  Gas  Press 

Relay 
Gen.  Brake  Air  Low  Pressure 
Unit  Cooling  Water  Relief  Valve 
Unit  Bearing  Cooling  Water  Strainer 
Air  Cooler  Header  Water  Low  Pressure 
Gen.  Air  Cooler  Water  Low  Flow 
Gen.  .'\ir  Cooler  High  Temp 
Gen.  Differential 
Gen.  Differential  &  CO2  Release 
Gen.  Stator  Ground 
Gen.  Field  Ground 
Reduced  Freq.  Gen.  Stator  Ground* 
Gen.  Loss  of  Field 
Gen.  Overload 
Under  Frequency 
Reverse  Phase  Sequence  t 
Incomplete  Starting  Sequence 
Stator  Winding  RTD  High  Temp 
Gen.  Field  RTD  High  Temp 
Neg.  Sequence 

Loss  of  Instrument  Potential 
Synchronizing  DC  Failure 
Automatic  Synchronizing  DC  Failure 
Starting  Sequence  DC  Failure 
Protective  Relaying  DC  Failure 
Overvoltage  Frequency  Compensated 


Voltage  Restrained  Overcurrent 

Shaft  Deflection 

Unit  Creeping 

Unit  Overspeed 

Unit  Partial  Shutdown 

Unit  Shutdown  and  Lockout 

Unit  Cooling  Water  Transfer 

Grease  Pump  Failure 

W'icket  Gate  Shear  Pin  Failure 

Gen.  Over  Excitation 

Regulator  Overload 

Exciter  DC  Failure 

Interlocking  DC  Failure! 

Runner  Band  Drain  Valve  Failed  to  Close 

Runner  Band  Drain  Valve  Failed  to  Open 

Transfer  to  Manual  Control 

.^uto  Starting  (Master  Start) 

Auto  Start  Sequence  Complete 

Auto  Stopping 

Generator  Operation 

Reduced  Freq.  Operation* 

Synchronous  Condenser  Operation 

Motor  Operation  t 

Reduced  Freq.  Partial  Trip  Relay 

Local  Manual  Load  Control  On 

Remote — .Auto  Load  Control  On 

Voltage  Regulator  On 

Voltage  Regulator  Trip 

Wicket  Gate  Open 

lixciter  Field  Breaker  Open 

Exciter  Field  Breaker  Close 

Reverse  Exciter  Breaker  Opent 

Reverse  Exciter  Breaker  Closef 

Turbine  Water  Depressed 

Penstock  Valve  Open 

Penstock  Valve  Close 

480V  Control  Center  Power  Transfer  (2) 


PSB  Distr.  Center  Power  Transfer 

Loss  of  AC  from  480V  Control  Center 

DC  System  Failure 

Station  Service  Brkr.  Trip 

Station  Service  Transformer  Diff.  (4) 

Station  Service  Transformer  High  Temp  (4) 

Switchgear  Ground 

Main  Transformers  Emere.  Cooling  Water 

(2) 
Transformer  Emerg.  Cooling  Water  L^w 

Flow  (2) 
Transformer  Cooling  Water  Relief  Valve  (2) 
Gov.  Air  Revr.  Low  Air  Press 
Instrument  Air  Low  Press 
Control  and  Brake  Air  Header  Low  Press 

•Units  I,  3.  and  5  only 
t  Units  2.  4,  and  6  only 
:  Units  2  and  4  only 


NONUNITIZED  POINTS 

Sewage  System  Failure 

Brake  and  Control  Air  Comp.  Cooling  Water 
Low  Press 

Depressing  and  Service  Air  System  Low  Press 

Depressing  &  Service  Air  Comp.  Disch.  Air 
Temp  (2) 

Gov.  .Mr  Comp.  Cooling  Water  Low  Press  (2) 

Standby  Cooling  Water  Main  Valve  Activat- 
ed (2) 

Service  Air  Comp.  Cooling  Water  Low  Press 
(2) 

Chiller  Cond.  Cooling  Water  Strainer  Diff. 
Press 

Dewatering  Sump  High  Level 

Drainage  Sump  High  Level 


Dewatering  Gallery  Sump  High  Level 

Powerhouse  Oil  Room  CO;  Release 

Smoke  Detection  Equipment  Failure 

Return  .^ir  High  Temp 

Smoke  Detected 

230  kV  Cable  .'Mr  Temp 

River  Outlet  Valve  Gallery  Flooded 

River  Outlet  \"alve  Low  Oil  Press 

Outlet  Portal  Trashrack  Wrong  Position  (2) 

Circuit  Breaker  Trip  (4) 

Circuit  Breaker  Close  (4) 

480\'  Power  Transfer  (4) 

Light.  Distr.  Ctr.  Power  Transfer  (2) 

Pumps  Tripped  by  PCi&E 
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Table   13.     Alarms   and   Events  Monitored — Thermalito   Powerplant 


Thrust  Bearing  High  Oil  Level 

Thrust  Bearing  Oil  Temp 

Thrust  Bearing  Low  Oil  Press 

Thrust  Bearing  Low  Oil  Level 

Thrust  Bearing  High  Temp.  Relay 

Kaplan  Unit  Oil  Head  Lower  Reservoir  Low 

Level* 
Kaplan  Unit  Oil  Head  Excess  Leakage* 
Kaplan  Unit  Oil  Head  Upper  Reservoir  Low 

Level* 
Kaplan  Unit  Oil  Head  Upper  Packing  Box 

Temp* 
Kaplan  Unit  Oil  Head  Lower  Packing  Box 

Temp* 
AC  Bearing  Oil  Pump  Low  Oil  Flow 
Stator  Winding  High  Temp 
Gen.  Guide  Bearing  Temp  Relay 
Gen.  Bearing  Cooling  Water  Low  Flow 
Gen.  .Xir  Cooler  Header  Water  Low  Flow 
Unit  Cooling  Water  Low  Flow 
Upper  Runner  Seal  Water  Low  Flow 
Lower  Runner  Seal  Water  Low  Flowf 
Gov.  Sump  Tank  Low  Oil  Level 
Gov.  Low  Oil  Press 
Turbine  Packing  Box  High  Temp 
Turbine  Guide  Bearing  High  Oil  Level 
Turbine  Bearing  Disch.  Oil  Temp 
Turbine  Guide  Brg.  Cooling  Water  Low  Flow 
Turbine  Bearing  Oil  Supply  Low  Flow 


UNITIZED  POINTS,  TYPICAL  EACH  UNIT 

Turbine  Guide  Bearing  Temp  Relay 
Unit  Bearing  High  Temp 
Turbine  Guide  Bearing  Low  Oil  Level 
DC  Guide  Brg.  Oil  Pump  Start  Initiated 
-Main  Transformer  Sudden  Pressure 
Main  Transformer  Winding  Temp  Relay 
Main  Transformer  Winding  Temp 
Main  Transformer  Low  Oil  Level 
Main  Transformer  Oil  Temp 
.Main  Transformer  Tank  Press 
Main  Transformer  Hot 
Main  Transformer  Inert  Air 
Main  Transformer  Water  Spray 
Main  Transformer  DifT.  Auxiliary  Relay 
Main  Transformer  Fan  Supply  or  Control  Volt- 
age Low 
Overcurrent  Voltage  Restrained 
Overvoltage  Frequency  Compensated 
Reverse  Phase  Sequence! 
Gen.  Loss  of  Field 
Gen.  Field  Ground 
Gen.  CO2  Release 
Neg.  Sequence  Phase  Unbalance 
Incomplete  Starting  Sequence 
Directional  Overcurrent  Relay 
Under  Frequency  Relayt 
Unit  Partial  Shutdown 
Unit  Shutdown  and  Lockout 


Kmergency  Shutdown  Solenoid 

Gen.  Brake  Air  Press  Low 

(jen.  ."Mr  Cooler  Disch.  .Mr  Temp 

Grease  Pump  Failure 

Shaft  Deflection 

Unit  Creeping 

Unit  Overspeed 

(Sen.  Diff.  Auxiliary  Relay 

Runner  Band  Drain  Valve  Failed  toOpent 

Runner  Band  Drain  Valve  Failed  to  Closet 

Gov.  Pressure  Tank  Low  Oil  Level 

Automatic  Starting 

Automatic  Stopping 

Transfer  to  Manual  Control 

Generator  Operation 

Motor  Operationt 

Synchronous  Condenser  Operation 

Motor  ."Xuto  Start  Sequence  Completet 

Voltage  Regulator  Switch  Position  .Manual 

Voltage  Regulator  Switch  Position  .Auto 

Exciter  Field  Breaker  Closed 

Turbine  Water  Depressed 

Wicket  Gate  Open 

1.1.  KkV  Gen.  Circuit  Breaker  Trip  (4) 

13.8kV  Gen.  Circuit  Breaker  Close  (4) 

13.SkV  Motor  Circuit  Breaker  Tript  (3) 

.'\uto  Load  Control  On 

Kaplan  Sump  High  Water* 


Drainage  &  Dewatering  Pumps  Controller 

Low  Air  Press 
Control  Air  Low  Pressure  (4) 
Powerplant  Oil  Room  CO2  Release 
DC  Bus  Ground  or  Low  Voltage 
Loss  of  Essential  AC 

Depressing  &  Service  Air  System  Low  Press 
Depressing  &  Service  A\t  Compr.  Disch.  .Mr 

Temp  (3) 
Dewatering  Sump  High  Water  Level 
Drainage  Sump  High  Water  Level  (2) 
Draft  Tube  Dewatering  Valve  Pit  Sump  High 

Water  (2) 


NONUNITIZED  POINTS 

High  Press  ,\ir  Compr.  Cooling  Water  Press 

(2) 
Service  Air  Compr.  Cooling  Water  Press  (3) 
Smoke  Detected 

Smoke  Detector  Equipment  Failure 
Control  Room  High  Air  Temp 
(Sencrator  Floor  High  Ait  Temp 
4S0V  Tic-Breaker  Transfer  (2) 
Circuit  Breaker  Trip  (6) 
Circuit  Breaker  Close  (6) 
PCB  Open  (2) 
PCB  Close  (2) 
PCB  Trip  (2) 


PCB  Lp  Svs  Lo  PSI 
PCB  Hp  Sys  Lo  PSI 
Pumps  Tripped  by  PG&l''. 
PG&I'".  Load  Shed  Sig  Loss 
230kV  Int  Sw  Trip  (4) 
.\trm  .'\ir  Comp  Run  (2) 
PCB  HpSvs  1.0  PSI 
SFt  Diff  Press  (2) 
PCB  Lp  Svs  Lo  PSI 
PC;&E  Standbv  L<.  Volts 
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Table  14.     Alarms   and    Events   Monitored — Oroville   Switchyard 


Ckt.  Brkr.  Low  Press  System  Low  Press 
Ckt.  Brkr.  High  Press  System  Low  Press 
Ckt.  Brkr.  Fault  Trip 


UNITIZED  POINTS,  TYPICAL  EACH  UNIT 

Ckt.  Brkr.  Trip 
Ckt.  Brkr.  Close 
230  kV  Cable  Pilot  Wire 


Unit  Disconnect  Switch  Open* 
Unit  Disconnect  Switch  Close* 


Ckt.  Brkr.  Trip  (3) 

Ckt.  Brkr.  Close 

Transformers  High  Temp 

Sta.  Serv.  Transformer  Oil  Temp 

Sta.  Serv.  Transformer  Differential 

Sta.  Serv.  Transformer  Sudden  Press 

Sta.  Serv.  Transformer  Winding  High  Temp 

Sta.  Serv.  Transformer  Main  Tank  Oil  Level 

Sta.  Serv.  Transformer  LTC  Tank  Oil  Level 

Sta.   Serv.   Transformer   Main  Tank   Press 

Relief 
Sta.  Serv.  Transformer  LTC  Tank  Press  Re- 
lief 
Sta.  Serv.  Transformer  Cylinder  Low  Press 
Sta.  Serv.  Transformer  Tank  Low  Press 
Sta.  Serv.  Transformer  Tank  High  Press 

•  Units  1,  3,  and  5  only 


NONUNITIZED  POINTS 

Sta.  Serv.  Transformer  Winding  Temp 

230  kV  Cable  System  Trouble 

Intake  Gate  Trouble  (2) 

Control  Bldg.  Oil  Storage  Room  CO:  Release 

Control  Bldg.  Sump  Pump 

Switchyard  DC  Circuit  Failure  (3) 

230  kV  Bus  Protective  Relaying  DC  Failure 

DC  System  Power  Failure 

Core  Block  Drainage  Pump  Running  (3) 

Ckt.  Brkr.  High  Press  System  Low  Press  (3) 

Ckt.  Brkr.  Low  Press  System  Low  Press  (3) 

Displav  Panel  Trouble 

Ckt.  Brkr.  Fault  Trip  (3) 

Ckt.  Brkr.  Trip  (3) 

Line  Disconnect  Switch  Close  (3) 

Main  Bus  Sect.  Switch  Open  (2) 


Main  Bus  Sect.  Switch  Close 

Bus  Section  Lockout  Relay  (3) 

Main  Bus  Undervoltage  (3) 

Communication  Channel  Reach  Failure  (3) 

Ckt.  Brkr.  Close  (3) 

Line  Disconnect  Switch  Open  (3) 

230  k\-  Cable  Gallery  High  .Air  Temp 

Trashrack  Raised  on  Pumping 

Sewage  System  Failure 

Fallout  Condition 

Pumping  .Mode  of  Operation 

Generating  Mode  of  Operation 

Power  Canal  Level  High 

Power  Canal  Level  Low 

Load  Control  Suspended 


responsibility  of  the  ACC  operator  to  ensure  that 
sufficient  generating  capacity  is  available  to  the 
load-control  subsystem  to  cover  scheduled  ADS 
generation.  This  is  accomplished  by  starting  units, 
bringing  them  to  speed-no-load,  and  switching  them 
into  the  load-control  subsystem  either  manually  or 
through  the  supervisory  system. 

The  load-control  subsystem  can  also  be  operated 
manually  through  the  load-control  console  in  the 
absence  of  the  ADS  signal.  This  is  accomplished  by 
entering  block  generation  amounts  through  the 
schedule  setter  on  the  load-control  console.  Figure  28 
shows  the  automatic  load-control  console. 

In  the  generating  mode  of  operation,  the  load-con- 
trol console  matches  required  water  flow  at  Edward 
Hyatt  and  Thermalito  Powerplants.  The  console  also 
distributes  load  on  operating  units  in  both  Power- 
plants  in  the  most  efficient  manner  for  the  desired 
generating  load.  The  unit  controllers  compare  signals 
representing  actual  and  computed  unit  generating  re- 
quirements. The  output  signal  from  each  unit  control- 
ler is  fed  to  the  governor  speed  adjust  until  both  actual 
and  computer  signals  to  the  unit  controller  are  equal. 

In  the  pumping  mode  of  operation,  the  load-control 
console  adjusts  the  position  of  wicket  gates  of  the 
three  pumping  units  at  Hyatt  to  achieve  ma.ximum 
efficiency  at  actual  pumping  head.  Pumps  at  both 
Powerplants  are  started  manually  or  tripped  au- 
tomatically as  required  by  excessive  changes  in  the 
water  level  of  the  Thermalito  Power  Canal.  Unit  con- 
trollers compare  the  signals  representing  the  actual 
and  computed  positions  of  the  wicket  gates.  The  out- 
put signal  is  then  cut  off  and  no  further  control  takes 
place  until  the  pumping  head  at  Oroville  changes. 


Devil  Canyon  Control  System 

The  purpose  of  the  Devil  Canyon  Powerplant 
control  system  is  to  provide  local  automatic  and 
remote  control  of  the  plant  from  the  Southern 
California  ACC  or  from  the  POCC. 

Local  Automatic  Control.  The  local  automatic 
control  system  is  provided  to  (1)  regulate  the  turbine 
power  output;  (2)  maintain  the  Devil  Canyon 
Afterbay  level  within  designated  limits,  and  (3) 
match,  insofar  as  possible,  turbine  flow  to  afterbay 
outflow.  Major  components  of  the  control  system  are 
the  afterbay  water-level  sensor,  a  control  switchboard 
with  associated  metering,  and  the  governor 
speed-adjust  transducer. 

The  control  switchboard,  located  in  the  plant 
control  room,  contains  setpoint  devices  and  meters 
and  indicating  meters  for  total  afterbay  flow,  total 
plant  flow,  unit  flow,  and  afterbay  water  level. 
Recorders  are  provided  for  afterbay  water  level,  total 
water  flow,  and  unit  flow. 

The  local  automatic  control  system  was  designed  to 
coordinate  operation  of  the  turbines  and  Afterbay 
under  the  following  conditions  (operating  modes): 

1.  Turbine  flow  matches  total  afterbay  outflow, 
with  corresponding  varying  power  output. 

2.  Turbine  flow  matches  total  afterbay  outflow 
within  minimum  and  maximum  flow  limits  (variable 
power  output  within  minimum  and  maximum 
limits). 

3.  Constant  turbine  flow  independent  of  total 
afterbay  outflow  (constant  power  output). 

The  control  system  adjusts  impulse  turbine  needles 
by   providing  a   variable,   direct-current,  command 
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signal  to  the  governor  speed-adjust  transducer.  A 
0-volt  signal  corresponds  to  the  speed-no-load  (SNL) 
needle  position,  and  a  1-volt  signal  represents  the  fully 
open  (or  100%)  needle  position.  The  needles  are 
linearly  positioned  from  the  SNL  position  to  100% 
open  for  voltages  between  0-  and  1-volt  DC.  The 
speed-adjust  transducer  adjusts  the  needles  by  direct 
modulation  of  the  governor  pilot  valve  cage.  This 
modulation  is  provided,  in  addition  to  the  normal 
pilot  valve  cage  movement  caused  by  fly-ball  action,  in 
response  to  load  variations  in  the  utility  system. 

The  control  system  is  regulated  through  adjustment 
of  four  setpoints,  either  locally  from  the  control 
switchboard  or  remotely.  These  setpoints  are: 

1.  Afterbay  water  level,  in  feet 

2.  Minimum  needle  position,  in  percent 

3.  Maximum  needle  position,  in  percent 

4.  Controller  gain,  in  percent  of  needle  stroke  per 
'/j-foot  change  in  afterbay  level. 

The  control  system  is  capable  of  matching  the 
steady-state  turbine  flow  to  total  afterbay  outflow 
provided  total  afterbay  outflow  remains  within  the 
range  between  the  minimum  needle  position  and 
maximum  needle  position  setpoints. 

When  afterbay  water  level  is  above  afterbay 
water-level  setpoint,  turbine  needles  will  be  at  the 
position  selected  as  minimum  needle  position 
setpoint.  As  afterbay  water  level  drops  below  setpoint 
level,  the  control  system  will  cause  the  turbine  needles 
to  open  farther  at  a  rate  determined  by  the  controller 


gain  setpoint.  If  sustained  afterbay  outflow  is  in  excess 
of  turbine  flow  selected  as  the  maximum  needle 
position  setpoint,  the  Afterbay  will  drain  unless 
setting  of  the  maximum  needle  position  setpoint  is 
increased  or  outflow  is  reduced.  If  sustained  afterbay 
outflow  is  less  than  turbine  flow  selected  as  the 
minimum  needle  position  setpoint,  the  Afterbay  will 
be  overtopped  and  excess  flows  will  be  discharged 
through  the  spillway. 

The  minimum  needle  position  setpoint  ensures  that 
power  output  of  the  plant  does  not  drop  below  a 
scheduled  amount. 

Remote  Control.  A  small  computer  system  (an 
8K,  Honeywell  H-316)  is  used  at  the  plant  to  provide 
the  interface  with  the  aqueduct  control  system  for 
remote  control.  This  computer  is  interfaced  to  the 
local  automatic  control  system  and  to  a  standard 
communications  system  at  the  plant.  An  operator  at 
the  ACC  has  command,  data,  status,  and  alarm 
capabilities  to  the  plant  as  shown  in  Table  15. 

Operation  of  the  Devil  Canyon  Afterbay  complex  is 
closely  tied  to  operation  of  the  plant.  Direct-wire 
operation  of  the  flow  gates  controlling  outflow  from 
the  Afterbay  is  possible  from  the  Devil  Canyon 
Powerplant.  In  addition,  complete  remote  monitoring 
and  control  of  the  Afterbay  is  possible  through  a 
separate  small  computer  system  installed  in  the 
afterbay  control  house.  An  operator  at  the  ACC  has 
control  and  data  monitoring  capability  of  the 
Afterbay  as  shown  in  Table  16. 
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Figure   28.      Automatic    Load    Control    Console 
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Table   15.      Information   Exchanged   Between  the  Devil   Canyon   Powerplant  System  and  the  ACC 


COMMANDS 

Telephone  Audin  Sienal 

Unit  Start  (each  unit) 

Unit  Stop  Normal  (each  unit) 

Unit  Stop  Emergency  (each  unit) 

Unit  Gain  Setpoint  (each  unit) 

Afterbay  Water  Level  Setpoint 

Unit  Needle  Position  Max.  Setpoint  (each 

unit) 
Unit  Needle  Position  Min.   Setpoint   (each 

unit) 
Select  2,  4,  or  6  Nozzles  (each  unit) 
Emergency  Close  No.  1  and  No.  2  Headworks 

Valves 

DATA 

Afterbay  Water  Level 

Plant  Flow  Rate  (CFS) 

Main  Bus  Voltage 

Real  Power 

Vars  (each  unit) 

Afterbay  Water  Level  Setpoint 

Unit  Needle  Position  Max.  Setpoint  (each 

unit) 
Unit  Needle  Position   Min.   Setpoint  (each 

unit) 
Unit  Gain  Setpoint  (each  unit) 
Plant  Energy  (kWh) 


STATUS 

Plant 

Headworks  Valve  No.  1  Closed 

Headworks  Valve  No.  2  Closed 

Unit  ON 

Unit  ON 

Unit  ON 
Unit  (each  unit) 

Unit  under  Local  Control 

Unit  under  Area  Control  Center 

Unit  I/P  (in  parallel) 

Unit  S/D  (shutdown) 

Unit  0/S  (out-of-scrvice) 

Turbine  ShutofT  Valve  Closed 

6  Nozzles  in  use 

4  Nozzles  in  use 

2  Nozzles  in  use 


ALARMS 

Plant 

Building  Entry 

Plant  Warning 

Telco  Equipment  Failure 

Transmission  Line  Breaker  Tripped 

Plant  Local  Control 

Primary  Power  Failure 


Drainage  Sump  High 

Main  Bus  Undervoltage 

Transformer  Trouble 

Main  Breaker  Lockout 

Hydraulic  System  Trouble 

Water  Level  Controller  Failure 

Warning  Headworks  Valves — Gales  Clos- 
ing 

Headworks  Primary  Power  Failure 

Headworks     Valves — Gates     Emergency 
Closure  Activated 

Headworks  Trouble 
Unit  (each  unit) 

Unit  Lockout 

Valve  Trouble 

Turbine  Bearing  Temperature  High 

Start  Failure 

Electrical  Fault 

Unit  Creep 

Generator  Stator  Temperature  High 

Vibration  High  (Shaft  Deflection) 

Overs  peed 

Generator  Bearing  Temperature  High 


Table  16.      Information  Exchanged   Between  the  Devil  Canyon  Afterbay  System  and  the  ACC 


COMMANDS 

Manual  Control      \  c     q     .     \       \;  n      n-     r 

A,,. ,-■    r-  „.     1  r  'or  oanta  Ana  Valley  Pipelme 

Automatic  Lontrolj  '       ^ 

Telephone  Audio  Signal 

Flow  Rate  Setpoint  for  Santa  Ana  Valley  Pipeline 

Gate  Position  Reference  (each  turnout— 4  total) 


DATA 

Flow  Rate  Setpoint  for  Santa  Ana  Valley  Pipeline 
Flow  Rate  in  Santa  Ana  Valley  Pipeline 
Gate  Position  Reference  (each  turnout) 
Gate  Position  (each  turnout) 


STATUS 

Manual  Control      1  t      c     .     a       \t  n       d-     i- 
Automatic  Control/  ^°'  ^anta  Ana  Valley  P.pehne 

Gate  Closed  (each  turnout) 

ALARMS 

Loss  of  Signal  CI  Line 
Loss  of  Signal  DSA  Line 
Telco  Equipment  Failure 
Building  Entry 
Remote  Control  Lockout 
Computer  Memory  Parity  Error 
Computer  Failure 
Computer  High  Temperature 
Gate  Failure  (each  turnout) 
Gate  Lockout  (each  turnout) 


Design  of  Aqueduct  Area  Control  Centers 

The  concept  of  five  area  control  centers,  their 
locations,  and  the  area  of  the  Project  that  each  controls 
is  described  in  Chapter  IV,  "Operations  Control 
Plan."  The  Oroville  Area  Control  Center  is  an 
integral  part  of  the  design  of  the  Oroville-Thermalito 
control  system.  The  remaining  four  aqueduct  area 
control  centers  are  functionally  similar  and  are 
described  herein. 

Functional  Description 

Each  ACC  is  the  master  control  station  of  an  area  of 


the  State  Water  Project.  It  shares  this  status  with  the 
POCC  in  Sacramento  which  can  assume  the  function 
of  an  ACC  (except  the  Oroville  ACC)  in  a  back-up 
mode  whenever  jurisdiction  is  assumed  by  the  POCC. 
The  four  Aqueduct  ACCs  and  the  POCC  have 
compatible  communications  and  are  integrated  into 
one  overall  control  system. 

The  major  components  of  an  ACC  are  as  follows: 

1.  Computer  System 

2.  Communications  Equipment 

3.  Operator's  Control  Console 

Figure  29  is  a  block  diagram  of  a  typical  ACC. 
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Figure   29.      Fundionol    Block    Diagram— Typical    Area    Control    Center 
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Computer  System 

The  computer  systems  vary  in  size  from  24K  to  32K 
words  of  16-bit  core  memory.  All  have  secondary 
storage  in  the  form  of  a  magnetic  disk  or  drum.  The 
amountof  secondary  storage  varies  from  262K  to  512K 
words  (16  bit).  The  computers  are  of  the  type 
designated  as  general-purpose  mini-computers  as 
manufactured  during  the  years  1968-1972. 

The  computers  utilized  in  the  Delta,  San  Luis,  and 
San  Joaquin  ACCs  are  Hewlett-Packard  Model  2116B. 
The  computer  utilized  in  the  Southern  ACC  is  a 
Honeywell  Model  316. 

Standard  peripheral  equipment  is  provided  includ- 
ing a  paper  tape  reader  and  punch  and  a  teletype  in- 
put/output device.  This  peripheral  equipment  is  used 
for  programming  purposes  only.  Nonstandard  pe- 
ripheral equipment  includes  special-purpose  equip- 
ment required  to  interface  to  the  communications 
equipment,  the  operator's  control  console,  and  the  ju- 
risdiction control  system.  The  computer  system  and 
all  other  control  system  equipment  in  the  ACC  are 
powered  by  an  uninterruptible  power  supply. 

The  functions  of  the  computer  system  can  be 
categorized  as  data  acquisition,  information  display, 


and  command  execution.  Data  acquisition  is  a  con- 
tinual automatic  function.  In  a  polled  communication 
system,  each  piece  of  information  transmitted  from  a 
remote  site  to  the  ACC  is  transmitted  as  a  result  of  a 
specific  request  (interrogation)  issued  from  the  ACC. 
These  interrogations  are  continually  being  transmit- 
ted from  the  ACC  at  a  rate  of  about  eight  per  second. 
The  returning  data  are  validated  by  the  ACC  com- 
puter and  stored  in  memory  for  future  use.  The  com- 
puter also  performs  certain  arithmetic  calculations  on 
some  of  the  data  (e.g.,  flow  rates  through  check  struc- 
tures, calculated  from  differential  head  and  gate  posi- 
tion) and  stores  the  results  for  future  use.  Data  are 
interrogated  (scanned)  at  ma.ximum  intervals  of  ap- 
proximately two  minutes  with  some  data  (e.g.,  alarms, 
certain  status  information,  and  certain  power  parame- 
ters) being  scanned  every  15  seconds.  Thus,  the  most 
current  information  is  always  available  to  the  opera- 
tor. The  computer  also  formats  these  data  for  presen- 
tation to  the  operator  through  various  devices  of  the 
operator's  control  console.  For  control,  the  computer 
accepts  commands  from  the  operator  through  the  op- 
erator's control  console  for  immediate  or  later  execu- 
tion. 

If  the  POCC  is  in  control  of  the  area  (as  determined 
by    the    jurisdiction    control    system),    the    ACC 


Figure   30.     Typicol   Operator's   Console — Area   Control   Center 
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Figure  31.     Typical    Con 


ole — Area    Control    Center 


computer  still  performs  the  data  acquisition  function 
by  accepting  and  storing  incoming  data  received  in 
response  to  the  POCC  scanning  and  makes  these  data 
available  to  the  operator.  The  ACC  computer  also 
inhibits  interrogations  and  commands  from  being 
transmitted  from  the  ACC  during  the  periods  of  time 
the  POCC  is  in  control. 

Communications  Equipment 

The  communications  equipment  in  the  ACC  is 
functionally  similar  to  the  standard  communication 
systems  at  the  remote  sites.  The  modem  transmitters 
at  the  ACC  continuously  transmit  a  signal  to  the 
communication  line.  During  the  period  between 
outgoing  interrogation  messages,  a  1,200  hertz 
"carrier"  tone  is  applied  to  the  line.  The  purpose  of 
this  signal  is  to  keep  any  noise  on  the  communication 
line  from  being  interpreted  by  the  remote  site 
receivers  as  valid  data  signals. 

The  communication  terminal  equipment  at  the 
ACC  is  the  simplest  of  all  terminal  equipment  installa- 
tions. It  does  not  have  the  enhancements  of  the  remote 
sites  but  merely  performs  "sync"  recognition /genera- 
tion and  serial-to-parallel  conversion  of  the  received 
messages. 

Each  ACC  is  designed  for  the  use  of  two  communi- 
cation systems,  with  each  having  capability  for  both 
an  "A"  and  "B"  communication  line.  The  only  ACC 
at  which  both  communication  systems  are  in  use  is  the 


San  Joaquin.  At  the  other  ACCs,  the  second  system  is 
provided  for  future  expansion. 

Operator's  Control  Console 

The  operator's  control  console  is  the  interface  de- 
vice between  the  operator  and  the  computer  system. 
The  console  is  a  two-station,  sit-down,  control  console 
containing  the  following  principal  functional  compo- 
nents: 

1.  Cathode  Ray  Tube  (CRT)  display  and  control 
system 

2.  Alarm  display  panel 

3.  Console  printout  device 

Figure  30  shows  the  layout  of  the  operator's  console. 

Facing  the  operator's  console  is  a  communication 
console  containing  a  mobile  radio  terminal  and  other 
voice  communications  equipment.  Figure  31  shows 
this  console. 

CRT  System.  Monitoring  and  controlling  of 
remote  sites  from  the  ACC  are  accomplished  through 
the  CRT  system.  This  is  the  main  operator  input/ 
output  device  to  the  computer.  Figure  32  shows  a  typi- 
cal CRT  station  layout. 

To  display  information,  the  operator  calls  up  CRT 
"pages"  of  information  by  use  of  the  "page"  keyboard 
matrix  (Figure  33).  Depressing  these  keys  interrupts 
the  computer,  which  in  turn  causes  the  requested  page 
containing  the  latest  data  to  be  displayed  to  the 
operator  over  the  appropriate  CRT  device. 
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Figure  32.     Typical    CRT    Station — Area    Control    Center    Operator's    Console 


Figure  33.     Typical    Page    Keyboord    Matrix — Areo    Control    Center    Operotor's   Console 
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These  pages  are  also  used  by  the  operator  to  execute 
commands  to  the  system.  If  the  operator  calls  the  page 
in  the  "edit"  mode,  the  page  will  be  displayed  with 
"underline"  characters  positioned  in  variable  loca- 
tions into  which  the  operator  can  enter  information. 
Figure  34  shows  a  check  structure  page  displayed  in 
the  edit  mode.  By  positioning  a  cursor  using  the  cur- 
sor positioning  controls,  the  operator  can  enter  com- 
mands and/or  numeric  characters  as  appropriate  in 
the  NEXT  COMM,  REF,  or  TIME  columns  of  the 
CRT  page.  When  he  has  completed  entering  the  infor- 
mation, the  page  can  be  executed,  causing  it  to  be 
transmitted  to  the  computer.  The  computer  validates 
the  commands  and  stores  them  for  transmission  to 
remote  sites  as  determined  by  the  entry  in  the  TIME 
column.  Using  this  capability,  the  operator  can  "pre- 
program" many  commands  for  automatic  execution 
by  the  computer  at  the  appropriate  time  of  day.  Figure 
35  shows  the  editing  capability  available  to  the  opera- 
tor. 

Pumping  plants  are  controlled  in  a  like  manner. 
Figure  36  shows  a  typical  pumping  plant  CRT  page. 

The  operator  can  request  display  of  "nonedit" 
summary  pages.  These  display  the  same  operating 
parameters  at  all  sites  within  the  control  area  but 
cannot  be  edited  for  command  entries.  Figure  37  is  a 
typical  summary  page  of  information. 


There  is  a  complete  set  of  page  keys,  cursor 
positioning  keys,  and  command  entry  keys  for  each  of 
the  two  operating  stations. 

Alarm  Display  Panel.  A  single-alarm  display 
panel  is  common  to  both  operating  positions  of  the 
console.  Figure  38  shows  the  alarm  panel.  This  panel 
annunciates  each  alarm  as  it  is  received  by  the  ACC 
computer  by  flashing  the  appropriate  site  window. 
The  operator  acknowledges  the  alarm  and  causes  the 
specific  alarm  conditions  to  be  displayed  on  the  panel 
by  use  of  the  alarm  panel  detail  keyboard  matrix 
which  is  located  between  the  two  operating  positions 
and  common  to  both.  Figure  39  shows  this  keyboard 
matrix. 

In  addition,  alarms  are  displayed  on  the  appropriate 
CRT  page  and  are  typed  out  over  the  console  printer. 
Additional  alarms  are  provided  on  the  console  con- 
cerning the  jurisdiction  system  and  computer  periph- 
eral equipment. 

Console  Printer.  The  operator's  console  is 
provided  with  a  printer  which  automatically  logs  all 
alarms  and  changes  in  status.  It  also  is  used  to  log 
certain  operating  data  on  operator  demand. 


Figure  34.     Check   Strudure   CRT   Page  in   "Edit"   Mode 
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Figure  35.     CRT   Editing   Capability 
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Figure  36.      Pumping   Plant   CRT   Page    Shown    in    the   "Edit"   Mode 
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Figure  37.      Summary    CRT    Page 


Figure  38.      Typical    Alarm    Panel — Area   Control    Center   Operator's    Console 


Figure  39.     Alarm  Panel  Keyboard  Matrix — Area  Control  Center  Operator's  Console 
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Jurisdiction  Control 

Normally,  control  of  the  Project  is  carried  out  from 
the  ACCs.  In  the  event  an  aqueduct  ACC  is  unable  to 
carry  out  this  control,  a  back-up  control  capability  is 
provided  from  the  POCC.  The  function  of  the  juris- 
diction control  system  is  to  ensure  that  equipment  in 
the  ACC  is  completely  disconnected  from  the  com- 
munication line  during  the  periods  the  POCC  is  in 
control.  This  system  is  described  in  detail  in  the  sec- 
tion "Design  of  the  POCC". 

Emergency  Power 

Each  ACC  is  provided  with  power  from  an 
uninterruptible  power  supply  consisting  of  a  static 
inverter  with  suitable  battery  supply.  The  batteries 
are  continually  charged  with  either  primary 
commercial  power  or  an  emergency  engine  generator. 

Design  of  the  Project  Operations  Control  Center 
The  Project  Operations  Control  Center  in  Sacra- 


mento serves  as  the  central  point  for  coordination  and 
direction  of  all  water  and  power  operation  of  the  State 
Water  Project.  It  provides  a  back-up  control  to  the 
aqueduct  area  control  centers  when  the  ACCs  are  out 
of  service  for  either  preventive  or  corrective  mainte- 
nance. Also,  it  provides  for  certain  automatic,  pro- 
jectwide,  computerized  reactions  to  emergency  condi- 
tions affecting  project  operations.  Figure  40  shows  the 
facilities  of  the  POCC. 

The    POCC    consists    of    the    following    major 
components: 

1.  Computer  System 

2.  Wall  Displays 

3.  Dispatcher's  Console 

4.  Cathode  Ray  Tube  (CRT)  System 

5.  Printers 

6.  Communications 

Figure    41    is    a    block    diagram    of    the    POCC 
equipment. 


Figure  40.      Project    Operations   Control    Center 
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Computer  System 

The  computer  system  is  the  main  element  in  the 
POCC.  The  computer  is  a  UNIVAC  418-11, 
general-purpose,  digital  computer.  It  is  the  same 
computer  used  in  the  South  Bay  Model  study 
(Chapter  III)  but  expanded  to  meet  the  needs  of  the 
POCC. 

System  components  are  a  central  processor  with 
6.^K  words  (18  bit)  of  core  memory;  three  drum 
memories,  one  of  which  is  6SK  words  and  two  of 
which  are  262K  words  each;  a  card  reader-line  printer; 
and  four  magnetic  tape  units  with  controller.  Figure 
42  shows  this  computer  system. 

The  computer  system  is  interfaced  to  the  wall  dis- 
plays, CRT  equipment,  printers,  and  communications 
system.  It  has  been  programmed  by  department  per- 
sonnel to  provide  all  necessary  capability. 

Wall  Displays 

A  wall  display  panel  provides  continuously  updated 
information  on  alarms  and  selected  status  and  data  for 
the  Project.  This  panel  is  a  floor-to-ceiling  display 
built  on  a  curve  with  a  total  length  of  approximately 
52  feet.  The  left  16  feet  of  panel  are  constructed  from 
standard  perforated  power  dispatcher's  board.  It  was 
supplied  blank  and  has  been  labeled  by  the  dispatchers 
to  reflect  the  single-line  diagrams  of  the  various 
pumping  and  power  plants  in  the  Project  and  the 
power  transmission  lines  and  substations  feeding 
them.  Lamps,  which  are  manually  lighted,  indicate  the 
status  of  switchyard  and  transmission  line  breakers. 
Figure  43  shows  this  portion  of  the  panel. 


To  the  right  of  these  panels,  the  display  panel  is 
constructed  of  plywood  with  a  single-line  profile  of 
the  Project  silk-screened  onto  the  face.  Beginning  on 
the  left  end  of  this  portion  of  panel  with  the 
Oroville-Thermalito  complex,  the  profile  extends 
"down  project"  to  Lake  Perris  on  the  right  end  of  the 
panel. 

Each  major  feature  of  the  Project  is  appropriately 
designated  with  a  special  symbol  and  is  equipped  with 
an  alarm  indicator.  In  some  cases,  special  rear-projec- 
tion digital  readouts  display  data  and  status  informa- 
tion. Figure  44  shows  the  finished  layout  of  the  wall 
display  panel. 

The  wall  display  is  updated  by  the  POCC  computer 
using  a  serial,  4,800-bit-per-second,  output  line.  The 
computer  formats  a  special  message  containing  a 
device  identification  (alarm  indicator  or  digital 
readout)  and  other  pertinent  data.  This  message  is 
received  by  the  wall  display  electronics  where  it  is 
decoded  and  routed  to  the  proper  location  on  the 
panel. 

Dispatcher's  Console 

1  here  are  four  operating  positions  for  the  dispatch- 
ers in  the  POCC.  The  two  shift  dispatchers  operate 
from  a  modified,  double-horseshoe  console  which  pro- 
vides surface  space  for  the  CRT  equipment,  typewrit- 
ers, calculators,  voice  communications  equipment, 
and  writing  areas.  Storage  space  within  the  console 
also  is  provided  for  reference  and  operating  docu- 
ments. 


figure  42.      Project   Operations   Control   Center   Computer   Systen 
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Figure  43.      Eledricol    Single    Line    Display    Ponel — Project   Operations    Control    Center 
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Figure  44.     Graphic    Display — Project    Operations    Control    Center 


CRT  System 

The  CRT  system  consists  of  eight  terminals 
coupled  to  the  computer  through  a  CRT  Multiplexer 
via  a  9,600-bit-per-second  input/output  (I/O)  line. 
The  CRTs  are  the  main  dispatcher  I/O  device  to  the 
computer  for  monitoring  and  control. 

Each  of  the  two  shift  dispatchers'  positions  have 
two  CRT  terminals  with  a  single  switchable  key- 
board. A  fifth  CRT  device,  an  alarm  and  events  dis- 
play without  keyboard,  is  centrally  located  between 
the  operating  positions  and  is  used  to  display  the  most 
recent  16  alarms  or  status  changes  of  the  facilities.  The 
sixth  CRT  terminal  is  located  on  the  programmed 


outage  dispatcher's  console,  the  seventh  CRT  is  locat- 
ed on  the  scheduling  dispatcher's  console,  and  the 
eighth  is  located  in  the  computer  room.  All  CR  T  ter- 
minals (except  for  the  alarm  CRT)  have  full  monitor- 
ing and  control  capability  and  can  be  completely 
interchanged  as  necessary. 

Operation  of  the  CRT  terminals  is  functionally 
similar  to  those  in  the  ACC.  An  exception  is  that  the 
functions  of  page  callup,  cursor  control,  and  com- 
mand and  data  entry,  which  are  accomplished  by  spe- 
cial keyboards  in  the  ACC,  are  performed  through  a 
standard  typewriter  keyboard  (alpha  numeric/nu- 
meric) at  the  POCC. 
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Printers 

Three  printers  are  located  behind  the  shift  dis- 
patcher's console.  One  printer  is  used  to  log  alarms  as 
they  are  received  by  the  POCC  computer;  the  second 
printer  is  used  to  log  changes  in  status  and  to  print  out 
the  dispatcher's  log  entries;  and  the  third  printer  is  an 
"off-line"  device  for  project  scheduling,  rescheduling, 
and  data  logging.  There  are  four  additional  printing 
devices,  one  located  in  each  of  the  aqueduct  ACCs. 
These  are  connected  to  the  POCC  computer  through 
leased  communications.  Dispatchers  can  direct  the 
same  information  to  these  printers  that  is  printed  out 
over  the  "off-line"  POCC  printer. 

Communications 

The  POCC  is  provided  with  both  voice  and  data 
communication  capabilities. 

Data  Communications.  Data  communications  to 
all  of  the  aqueduct  control  areas  are  provided  from  the 
POCC.  Modems,  compatible  with  those  used  in  the 
ACC,  are  provided  for  each  data  communications  line 
(both  primary  and  back-up).  The  receivers  at  the 
POCC  are  connected  to  the  communication  line  at  all 
times,  monitoring  data  responses  to  the  ACC 
interrogations  from  all  areas  simultaneously.  The 
transmitters  in  the  POCC  are  connected  to  the 
communication  lines  only  when  the  POCC  is 
providing  control  of  an  area. 

The  functions  of  the  communications  terminal 
equipment  at  the  ACC  are  performed  at  the  POCC  by 
the  communications  equipment  of  the  computer. 

The  Oroville  ACC  will  be  connected  in  the  future 
to  the  POCC  computer  for  data  monitoring  only  (no 
control)  by  a  slow-speed,  300-baud,  communication 
line. 

Other  data  communications  e.xist  between  the 
POCC  computer  and  the  Pacific  Gas  and  Electric 
(PG&E)  dispatch  center  in  San  Francisco  to  exchange 
load  data  and  power  availability  information. 
Additional  data  communications  are  planned  with  the 
U.S.  Bureau  of  Reclamation  (USBR)  and  other 
operating  agencies. 

Voice  Communications.  Voice  communications 
also  are  provided  in  the  POCC.  The  POCC  is 
connected  to  each  of  the  ACCs  with  a  separate, 
automatic,  ring-down  line.  It  is  also  connected  to  all 
the  aqueduct  ACCs  by  a  single,  manual-ring, 
party-line  arrangement  which  can  be  patched  across 
at  the  ACCs  to  give  the  dispatcher  voice  access  to  any 
remote  site  on  the  Project  via  the  voice  maintenance 
line. 

In  addition,  dedicated  voice  lines  are  provided  to 
other  agencies:  USBR,  PG&E,  Southern  California 
Edison  Company,  and  Los  Angeles  Department  of 
Water  and  Power. 

Operational  Capability 

Operational  capability  of  the  POCC  is  determined 


by  the  software  which  has  been  developed  for  the 
POCC  computer,  consistent  with  limitations  of  the 
POCC  equipment  itself.  Unlike  other  contracts  for 
control  systems  designed  by  the  Department,  where 
the  software  was  a  specification  requirement  imposed 
on  the  contractors,  the  application  software  for  the 
POCC  computer  was  designed  and  programmed  by 
department  personnel.  This  was  done  because  of  the 
long  time  frame  over  which  the  POCC  is  being 
implemented  and  the  many  different  systems  with 
which  the  POCC  must  be  compatible.  Software  for 
the  POCC  computer  can  be  simply  classified  as 
"on-line"  or  "off-line".  On-line  software  is  that  which 
is  required  for  data  monitoring  (DISPATCH)  and 
for  back-up  control  (BACS)  of  the  Project.  Off-line 
software  is  run  on  the  POCC  computer  but  in  a 
non-real-time  environment, 

DISPATCH  Software.  The  normal  mode  of 
operation  from  the  POCC  is  data  acquisition  and 
display.  The  POCC  computer  continuously  receives 
data  from  each  control  area  in  response  to 
interrogations  issued  from  the  ACCs.  These  data  are 
validated  and  stored  in  main  memory  for  use  by  other 
application  programs. 

Certain  of  these  data  are  automatically  directed  to 
the  wall  display  panel  whenever  the  data  is  different 
from  that  which  is  currently  displayed.  Certain 
alarms,  which  are  generated  by  the  POCC  computer 
by  comparing  incoming  data  against  dispatcher-en- 
tered high  and  low  limits,  are  also  directed  to  the 
display  panel.  These  limits  are  entered  through  the 
CRT  system.  By  proper  entry  of  these  limits,  the  com- 
puter will  "watch"  certain  key  operating  parameters 
in  the  Project  for  operating  exceptions.  Changes  in 
plant  unit  status  (I/S — in-service,  S/D — shutdown,  or 
O/S — out-of-service)  are  also  directed  to  the  display 
panel.  The  alarm  conditions  and  changes  in  unit  status 
are  also  transmitted  to  the  alarm  and  events  CRT  and 
to  the  appropriate  printer  for  a  permanent  record. 

Data  stored  in  main  memory  are  also  made  available 
to  the  dispatchers  on  demand  through  various  CRT 
devices.  By  calling  appropriate  CRT  pages,  the 
dispatcher  is  able  to  monitor  the  most  recent  data 
from  any  facility  of  the  Project.  Summary  pages 
containing  specially  formatted  data  displays  also  are 
available. 

Some  of  the  data  are  used  to  compute  operating 
parameters.  Flows  through  the  check  structures  are 
computed  from  the  differential  head  across  the  gates 
and  gate  positions.  Every  three  minutes,  the  project 
30-minute  integrated  power  demand  is  computed, 
projected  to  the  end  of  the  30-minute  demand  period, 
and  compared  against  the  schedule.  Deviations  arc 
alarmed  to  the  dispatcher. 

All  of  the  data  received  by  the  POCC  computer  are 
recorded  on  magnetic  tape  for  historical  analysis 
purposes. 

BACS  Software.  The  POCC  is  provided  with 
software  to  provide  remote  manual  control  of  the 
facilities  in  any  one  aqueduct  control  area  if,  for  any 
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reason,  its  ACC  is  out  of  service.  The  POCC  computer 
has  been  programmed  with  control  software  for  each 
of  the  four  aqueduct  control  areas.  This  software  is 
normally  stored  in  secondary  storage.  If  the 
dispatcher  desires  control  of  an  area,  he  can  do  so  by 
manually  initiating  a  BAGS  load  for  that  area.  From 
the  point  of  initiation,  the  load  and  initialization  of  the 
BAGS  software  is  completely  automatic.  The  POCG 
computer  loads  the  appropriate  BAGS  software  from 
secondary  storage.  Goncurrent  with  this,  the 
jurisdiction  control  system  sends  a  signal  to  the 
appropriate  AGG  equipment,  causing  the  AGG 
modem  transmitters  to  be  disconnected  from  the 
communication  line.  The  AGG  computer  is  switched 
"off  the  line",  and  an  acknowldgement  is  sent  back  to 
the  POCG.  Upon  receipt  of  the  acknowledgement,  the 
POGG  computer  initializes  the  BAGS  software  and 
notifies  the  dispatcher  that  the  POGG  is  in  control  of 
the  area.  The  system  is  interlocked  so  that  the  POGG 
can  be  in  remote  manual  control  of  only  one  area  at 
any  given  time.  When  BAGS  is  in  operation, 
DISPATGH  is  unaffected,  i.e.,  BAGS  is  in  addition  to, 
not  in  lieu  of,  normal  DISPATGH  operation. 

For  certain  special  conditions,  the  computer  itself 
can  automatically  assume  control  of  any  or  all  of  the 
aqueduct  control  areas.  These  special  conditions 
result  when  operating  conditions  dictate  the  need  for 
response  times  much  shorter  than  the  operators  and 
dispatchers  are  capable  of  achieving.  As  an  example, 
power  supply  contracts  for  interruptible  on-peak 
power  dictate  automatic  and  instantaneous  load 
dropping  upon  notification  from  the  suppliers  of 
certain  operating  conditions  on  their  system.  Within 
approximately  one  second  after  receipt  of  a  load  drop 
signal  from  the  PG&E  dispatch  computer  in  San 
Francisco,  the  POGG  computer  immediately  assumes 
control  of  all  areas  via  the  jurisdiction  control  system 
and  sends  emergency  unit  shutdown  commands  to  the 
appropriate  pumping  units  to  reduce  the  total 
pumping  load  of  the  Project.  Following  this  load  drop, 
the  POGG  computer  returns  control  back  to  the 
AGGs. 

Off-Line  Software.  The  POGG  computer  is  pro- 
vided with  the  capability  to  process  programs  in  a 
"background"  environment  concurrent  with  opera- 
tion of  the  DISPATGH  and  BAGS  software.  This 
background  or  off-line  software  is  related  to  the  sche- 

:       duling  and  rescheduling  of  project  operation,  data  log- 

I       ging,  and  report  generation. 

I  Whereas  the  DISPATCH  and  BAGS  software  have 

I  been  predominantly  written  in  assembly  language, 
the  off-line  software  has  been  written  in  FORTRAN. 
The  Department's  programming  staff  enhanced  the 
vendor-supplied  FORTRAN  to  provide  FORTRAN 
programmers  with  the  ability  to  access  the  real-time 
data  base  in  main  memory,  communicate  with  the 
CRT  system,  communicate  with  the  remote  printers 
in  the  AGGs,  and  utilize  the  secondary  storage  devices 
as  intermediate  data  storage  files.  This  allows  the  use 


of  practically  all  the  POCG  equipment  for 
FORTRAN  programs  concurrent  with  the  operation 
of  DISPATGH  and  BAGS  software. 

Jurisdiction  Control  System 

Coordination  of  control  from  the  POCG  or  the  ACC 
to  ensure  that  both  are  not  attempting  to  control  an 
area  simultaneously  is  the  function  of  the  jurisdiction 
control  system.  If  both  centers  were  to  attempt  con- 
trol at  the  same  time,  they  would  interfere  with  each 
other  on  the  command-interrogation  communication 
line  to  such  an  extent  that  coherent  control  or  interro- 
gation would  be  impossible  and  the  control  system 
would  become  inoperable. 

The  jurisdiction  system  was  implemented  to  pro- 
vide automatic  means  of  preventing  the  control  cen- 
ters from  interrogating  or  commanding  the  area 
simultaneously.  In  normal  operation,  system  jurisdic- 
tion is  controlled  at  the  POCG,  but  means  are  pro- 
vided tooverride  jurisdictional  the  ACC  if  the  normal 
system  fails. 

Figure  45  is  a  functional  block  diagram  of  the 
jurisdiction  system  used  at  the  POGG  and  each  area 
control  center.  Using  a  single,  full-duplex,  voice  grade 
communication  line  with  a  terminal  at  each  control 
center,  frequency-shift-keyed  audio  tones  are  used  by 
the  POGG  to  assign  jurisdiction  and  by  the  AGGs  to 
acknowledge  jurisdiction.  The  audio  tones  are 
"frequency  multiplexed",  that  is,  each  transmitted 
tone  is  within  a  band  120  hertz  wide  and  a  filter  on 
each  receiver  rejects  all  signals  outside  of  its  band. 
Thus,  the  band  for  tone  #7  is  centered  at  1,140  hertz 
(1,080  to  1,200  hertz),  while  tone  #8  is  centered  at 
1,260  hertz  (1,200  to  1,320  hertz).  Within  its  assigned 
band,  each  tone  has  three  states:  "Mark"  at  30  hertz 
above  center  frequency,  "idle"  at  center  frequency, 
and  "space"  at  30  hertz  below  center  frequency.  The 
receivers,  by  use  of  filters,  distinguish  these  states  and 
vary  their  output  to  the  computer  accordingly. 

In  normal  operation,  the  POCG  assumes  control  of 
one  or  more  areas  by  sending  a  "mark"  over  both  tone 
transmitters  to  the  ACC.  The  ACC  acknowledges 
POGG  control  by  returning  a  "mark"  over  both  its 
transmitters.  Area  control  center  jurisdiction  is 
similarly  established  by  an  exchange  of  "space" 
signals. 

Emergency  Power 

Power  to  the  equipment  in  the  POCG  will  be  pro- 
vided by  an  uninterruptible  power  supply  similar  to 
that  in  the  area  control  centers.  Emergency  power  is 
presently  being  provided  by  a  diesel  engine-generator. 

Design  of  Miscellaneous  Facilities 

In  addition  to  the  check  structure,  pumping  and 
power  plant  control  systems,  AGGs  and  the  POCC, 
other  miscellaneous  facilities  have  been  provided  with 
control  capability  as  a  part  of  the  Project's  control 
system  design  and  construction  activities. 
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Major  miscellaneous  systems  include: 

1.  Turnout  control  system 

2.  Structural     behavior     instrumentation 


data 


acquisition  system 

3.  Power  demand  monitor  system 
Another    major    system,    a    communication    line 
continuity  monitoring  system,  is  discussed  in  Chapter 
VII,  Communication   Systems.   In   addition,   minor 
miscellaneous  systems  have  been  provided. 

Turnout  Control  System 

A  turnout  is  a  structure  for  diversion  of  water  from 
the  Aqueduct  to  the  Project's  water  users.  These 
turnouts  can  be  either  gravity  flow,  pump  flow,  or  a 
combination  of  pump-gravity  flow. 

All  of  the  43  turnouts  in  the  San  Joaquin  control 
area  will  be  equipped  by  1975  with  systems  to  allow 
for  remote  monitoring  and  control  of  these  facilities 
from  the  San  Joaquin  ACC.  To  provide  this  remote 
capability,  each  turnout  is  being  provided  with  a 
turnout  control  electronic  assembly  (TCA). 

To  avoid  the  complications  created  by  increasing 
the  number  of  termination  points  on  aqueduct 
communication  lines,  turnout  monitoring  and 
controlling  data  will  be  relayed  through  nearby  check 
structures  as  subordinate  sites.  Interrogations  and 
commands  from  the  ACC  will  be  decoded  by  the 
check  structure  computer,  reformatted,  and 
transmitted  directly  to  the  turnout.  Returning  data, 
status,  and  alarms  will  be  treated  in  a  similar  manner. 

The  TCA  will  perform  the  following  functions  at 
the  turnout: 

1.  Scan  at  one-second  intervals  for  all  monitored 
data  and  store  data  for  transmission  when 
interrogated. 

2.  Operate  up  to  four  gates  to  cause  gate  position  to 
match  commanded  setpoints. 

3.  When  on  automatic,  adjust  gate  positions  to 
cause  flow  to  match  commanded  flow  setpoint. 

4.  Convert  analog  inputs  from  sensors  to  scaled 
digital  numbers. 

5.  Convert  scaled  digital  number  to  analog  output 
for  flow  controller  setpoint. 

The  TCA  will  perform  the  following  functions 
related  to  data  handling  to  and  from  the  check 
structure: 

1.  Receive,  demodulate,  check  for  validity,  and 
decode  commands  and  interrogations. 

2.  Encode;  add  parity  and  identification;  modulate; 
and  transmit  data,  status,  and  alarm  messages. 

The  design  of  the  TCA  incorporates  a 
microprocessor  system  to  perform  controlling  and 
monitoring  functions  at  the  turnout  and  to  provide 
communications  with  the  check  structure.  This 
microprocessor  system  will  be  provided  with  both 
PROM  (programmable  read-only  memory)  for 
program  storage  and  RAM  (random  access  memory) 
for  data  storage. 


Structural  Behavior  Instrumentation  Data  Acquisi 
tion  System 

To  monitor  the  structural  behavior  of  certain 
structures  during  seismic  activity,  the  Department 
has  installed  Structural  Behavior  Instrumentation 
Data  Acquisition  System  (SBIDAS)  equipment  at  A. 
D.  Edmonston,  Pearblossom,  and  Wheeler  Ridge 
Pumping  Plants;  Devil  Canyon  Powerplant;  and 
Castaic  Dam. 

The  purpose  of  this  SBIDAS  is  to  sample  output 
signals  from  various  types  of  transducers,  convert 
these  outputs  to  digital  signals,  and  record  these  sig- 
nals on  magnetic  tape  for  later  analyses.  The  transduc- 
ers monitored  consist  of  triaxial  accelerometers,  pore 
pressure  cells,  concrete  stress  meters,  and  soil  stress 
meters. 

The  SBIDAS  starts  operation  by  external  detection 
of  seismic  activity  greater  than  a  predetermined 
threshold  level.  Within  50  milliseconds  after  the 
threshold  value  has  been  exceeded,  the  SBIDAS  will 
begin  to  record  data  and  continue  to  record  for  a 
minimum  of  four  minutes,  or  until  the  seismic  activity 
is  reduced  below  the  threshold  level. 

The  SBIDAS  consists  of  a  mini-computer  system, 
input  signal  conditioning  networks,  input 
multiplexing  equipment,  analog-to-digital  conversion 
equipment,  and  a  magnetic  reel-to-reel  recording 
device. 

Procurement  and  installation  of  the  SBIDAS  were 
part  of  the  A.  D.  Edmonston  and  Pearblossom  Pump- 
ing Plant  control  system  contracts. 

Although  the  SBIDAS  was  designed  as  a  part  of  the 
control  system,  it  is  not  interfaced  to  the  remote 
control  system  in  any  manner.  It  is  a  local  device  and 
is  not  used  in  the  operation  of  the  Project. 

Power  Demand  Monitor  System 

Various  sources  of  electric  power  are  utilized  to 
meet  the  pumping  needs  of  the  Project.  The  total 
pumping  demand  cannot  exceed  the  total  of  power 
availability  without  creating  an  "extra  capacity" 
demand  on  the  supplier's  system.  This  extra  capacity 
results  in  heavy  penalties  if  the  demand  is  incurred 
during  an  on-peak  period. 

The  official  record  of  power  demand  for  each  of  the 
pumping  plants  is  recorded  on  magnetic  tape  record- 
ing equipment  at  each  plant.  These  tapes  are  assem- 
bled monthly  and  time  correlated,  and  projectwide 
demand  for  each  30-minute  interval  is  computed.  This 
occurs  substantially  after  the  extra  capacity  demand 
may  have  been  created.  To  reduce  the  possibility  that 
an  extra  capacity  demand  will  be  unknowingly  in- 
curred, the  Project  is  operated  with  a  15-megawatt 
margin  between  scheduled  power  and  actual  demand. 
This  margin  represents  an  investment  in  scheduled 
but  unused  power.  To  reduce  the  margin,  a  power 
demand  monitoring  system  is  being  implemented 
which  will  provide  the  dispatching  personnel  in  the 
POCC  with  real-time,  actual  integrated  demand  infor- 
mation as  it  occurs. 
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This  power  demand  monitoring  system  consists  of 
instrumentation  (demand  monitors)  on  each  of  the 
recording  demand  meters  at  the  plants.  These  demand 
monitors  will  be  scanned  every  1 5  seconds  by  the  ACC 
from  which  the  plant  is  controlled  and  the  data  will 
be  transmitted  to  the  POCC.  From  these  data,  the 
POCC  computer  will  compute  and  display  the 
Project's  30-minute  integrated  demand. 

Miscellaneous  Systems 

Other  miscellaneous  systems  have  been  designed  as 
a  part  of  the  Project's  control  systems.  These  include 


systems  for  control  and  monitoring  of  reservoir 
intakes  and  outlets,  pipelines,  flowmeters,  stream 
gauging  stations,  and  other  appurtenant  features. 

No  attempt  will  be  made  here  to  completely  de- 
scribe all  of  the  details  of  these  various  miscellaneous 
systems.  Most  of  these  systems  are  located  in  the 
Southern  California  control  area.  A  complete  descrip- 
tion of  the  control  and  monitoring  capability  existing 
at  each  of  these  facilities  can  be  found  in  Specification 
No.  71-01,  "California  Aqueduct  Control  System, 
Southern  California  Control  Area". 
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CHAPTER  VI.     CONTROL  SYSTEM  CONSTRUCTION 


All  significant  portions  of  the  work  to  construct 
control  systems  for  the  Project  were  performed  by 
contractors  under  provisions  of  the  State  Contract  Act 
(Government  Code  Sections  14250-14427).  Because 
of  the  requirement  of  this  Act  for  competitive, 
fixed-price  bidding,  work  progressed  through  the 
following  steps: 

1.  Prequalification  of  prospective  contractors. 

2.  Advertisement  and  award  of  the  contract. 

3.  Administration  of  contracts  and  supervision  of 
construction. 

Prequalification  of  Prospective  Contractors 

The  functional  nature  of  all  control  system  specifi- 
cations made  it  imperative  that  prospective  bidders  be 
thoroughlv  screened  for  technical  and  financial  capa- 
bility prior  to  participation  in  the  bidding  process.  A 
two-phase  prequalification  procedure  for  screening 
contractors  was  instituted  to  establish  lists  of  qualified 
bidders  whom  the  Department  confidently  believed 
could  satisfactorily  perform  work  if  awarded  to  them. 

In  Phase  I,  a  financial  ability  statement  (Reference 
7),  a  technical  qualification  questionnaire,  and  a 
two-page  quality  assurance  program  questionnaire 
(Reference  8)  had  to  be  completed  by  each 
prospective  contractor.  Only  those  contractors  whose 
technical  questionnaires  and  financial  ability 
statements  indicated  that  they  had  a  potential 
capability  of  performing  control  system  work  of  the 
complexity  and  magnitude  required  were  allowed  to 
enter  Phase  II  of  the  prequalification  procedure. 

Phase  II  consisted  of  an  inspection  of  the  prospec- 
tive contractor's  facilities  by  the  Department.  The  in- 
spection team  usually  consisted  of  representatives  of 
both  the  design  and  construction  organizations.  This 
team  filled  out  an  evaluation  guide  on  each  prospec- 
tive contractor.  The  contractor  was  judged  upon  his 
capability  of  being  the  prime  contractor  and  of  being 
the  actual  contractor  to  perform  the  work  on  any  or 
all  of  nine  subsystems.  The  nine  subsystems  were: 

1.  Site  equipment 

2.  Communication  terminal  equipment 

3.  Microwave  equipment 

4.  Data  collection  equipment 

5.  Computer 

6.  Control  consoles  and  displays 

7.  Recording  instruments 

8.  Load-control  equipment 

9.  Miscellaneous  structural  modifications 

Advertisement  and  Award  of  Contracts 

Construction  work  for  each  control  system  or  part 
thereof  was  advertised  as  set  forth  in  Article  3  of  the 
State  Contract  Act.  The  "Notice  to  Contractors"  (a 
document  which  generally  described  the  require- 
ments and  extent  of  the  work)  in  each  case  was  sent 


to  all  prospective  contractors  whose  prequalification 
indicated  they  were  capable  of  performing  the  work. 
Only  contractors  who  qualified  as  both  prime  contrac- 
tor and  as  contractor  for  one  or  more  subsystems  were 
allowed  to  bid  on  major  contract  specifications.  In 
many  cases,  the  Department  would  designate  a  specif- 
ic subsystem  for  which  all  bidding  contractors  must 
have  been  prequalified  for  performance  of  the  con- 
struction work.  This  was  done  where  a  specific  sub- 
system constituted  a  significant  portion  of  the 
proposed  work.  For  example,  pumping  plant  control 
system  contractors  were  required  to  be  prequalified  in 
the  computer  subsystem,  and  the  contractor  for  the 
Project  Operations  Control  Center  wall  display 
(Specification  No.  70-12)  had  to  be  prequalified  in  the 
control  consoles  and  displays  subsystem. 

Contracts  were  awarded  to  the  contractor 
submitting  the  lowest  responsible  bid.  Responsible 
bids  were  those  for  which  all  conditions  of  bidding 
stated  in  the  bidding  requirements  were  met  and 
which  were  determined  to  be  reasonable  in  cost  in 
comparison  with  the  engineer's  estimate. 

The  actual  bids  for  control  system  construction 
work  were  generally  much  lower  than  the  State's 
estimates.  The  total  of  all  bids  for  control  system  work 
was  almost  $10  million  less  than  the  $2.'>  million 
estimated  during  the  preliminary  design  phase.  This 
fact  probably  can  be  attributed  to  the  following 
factors: 

1.  A  decline  in  aerospace  industry  workload  at  the 
time  of  advertisement  for  control  system  construction 
work.  This  resulted  in  highly  competitive  bidding  by 
aerospace  companies  who  were  aggressively  seeking  a 
place  in  the  industrial  market. 

2.  \'arious  contractors  were  much  more  familiar 
with  essentially  cost-plus  federal  contracting  than 
with  the  provisions  of  the  State  Contract  Act  based  on 
fixed-price  bidding. 

3.  Decreasing  costs  in  the  1960s  of  equipment 
utilized  since  preliminary  estimates  were  made, 
because  of  advances  in  the  electronic  industry 
"state-of-the-art". 

Administration  of  Contracts  and  Supervision 
of  Construction 
The  Department's  standard  organization  for 
supervision  of  construction  activities  consisted  of 
Construction  Project  Offices  at  selected  locations 
throughout  the  State.  Each  Project  Office  was  staffed 
with  the  normal  complement  of  construction 
inspectors  experienced  in  civil,  electrical,  and 
mechanical  work.  A  Project  Office  was  responsible  for 
all  construction  within  a  particular  geographical  area. 

Electrical  and  mechanical  items,  such  as  turbines, 
pumps,  motors,  and  related  equipment,  which 
required  factory   inspection  and   which   were  to  be 
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installed  in  aqueduct  reaches  or  features  within  the 
Project  Offices'  areas  of  responsibility,  were  classified 
as  special  fixtures.  These  items  were  constructed 
under  furnish  and  install  contracts,  administered 
under  the  State  Contract  Act.  The  furnish  portion  of 
the  work  (design,  manufacturing,  and  factory  testing) 
which  took  place  at  the  contractor's  facility  for  these 
contracts  was  supervised  by  department  inspectors  at 
the  manufacturing  facility  under  supervision  of  con- 
struction forces  located  in  Sacramento.  Supervision  of 
the  installation  and  field  testing  work  was  performed 
by  construction  forces  of  the  appropriate  Project  Of- 
fice. 

Early  Control  System  Construction  Supervision 

Initially,  all  control  systems  were  considered  to  be 
in  the  category  of  special  fixtures  (factory  inspection 
with  field  installation).  Two  contracts  were  com- 
pleted under  this  concept  of  contract  administration 
and  construction  supervision:  the  Delta  Pumping 
Plant  control  system  (Specification  No.  65-41),  and 
the  interim  alarm  and  level  control  system  (Specifica- 
tion No.  67-32).  Also,  substantially  all  of  the  South 
Bay  Model  was  completed  in  this  manner. 

In  early  1968,  however,  technical  difficulties  arose 
on  the  South  Bay  Model  which  required  technical 
expertise  not  available  in  the  Project  Office.  This 
caused  the  Department  to  change  the  method  of  con- 
struction supervision  for  the  remainder  of  the 
contract.  A  resident  engineer  headquartered  in  Sacra- 
mento, a  specialist  in  control  systems  engineering, 
was  assigned  to  provide  technical  supervision  on  site 
for  the  remaining  work  on  the  South  Bay  Model.  Con- 
tract administration  and  overall  responsibility  for  the 
contract,  however,  remained  with  the  Project  Office 
within  its  delegated  authority  until  completion  of  the 
work. 

In  1966,  construction  began  on  the  Oroville-Ther- 
malito  control  system  (Specification  No.  66-44).  Ini- 
tially, this  system  was  treated  as  a  special  fixture.  In 
late  1967,  the  design  and  factory  test  work  for  this 
contract  had  not  yet  been  completed.  All  other  con- 
struction activities  for  which  the  Oroville  Project  Of- 
fice had  responsibility  were  nearing  completion.  It 
was  apparent  that  the  control  system  contract  would 
not  be  completed  until  after  the  date  the  Department 
had  established  for  officially  closing  the  Oroville 
Project  Office. 

As  a  result  of  this,  in  April  1968  the  Department 
changed  the  method  of  construction  supervision  and 
contract  administration  of  the  Oroville-Thermalito 
control  system  to  a  method  similar  to  that  which  had 
recently  been  established  for  completion  of  the  South 
Bay  Model.  A  control  system  engineer  was  assigned 
from  Sacramento  as  resident  engineer  to  supervise  the 
balance  of  the  construction  work  on  the  system.  Un- 
like the  South  Bay  Model,  however,  the  resident  engi- 
neer on  the  Oroville-Thermalito  control  system  was 


provided  with  contract  administration  support  from 
Sacramento  headquarters  contract  administration 
staff.  The  resident  engineer  was  assisted  in  field  instal- 
lation inspection  and  testing  work  by  control  system 
technicians  on  loan  from  the  Oroville  Field  Division, 
Division  of  Operations  and  Maintenance. 

Aqueduct  Control  System  Construction  Supervision 

As  a  result  of  the  experience  associated  with 
inspection  of  the  South  Bay  Model  as  a  special  fixture, 
it  was  decided  to  employ  a  different  method  of 
construction  supervision  for  the  aqueduct  control 
system  contracts.  The  method  used  was  the  same 
method  as  had  been  employed  to  complete  the 
Oroville-Thermalito  control  system.  This  method  of 
construction  supervision  (control  system  task  force  as 
it  came  to  be  called)  was  used  to  supervise  completion 
of  all  of  the  California  Aqueduct  control  systems  and 
most  of  the  facilities  in  the  Project  Operations  Control 
Center. 

A  resident  engineer,  a  specialist  in  control  system 
engineering,  was  assigned  to  each  contract  beginning 
with  preparation  of  in-house  designs  and 
specifications.  Personnel  with  experience  in  the 
inspection  of  electrical  or  other  construction  work 
were  assigned  to  inspect  the  electrical  and  mechanical 
activities  associated  with  the  control  system  in  the 
field.  Personnel  with  control  system  experience  were 
assigned  as  factory  inspectors  and  to  assist  in  field 
installation  and  checkout  of  the  equipment.  Other 
assistance  to  the  resident  engineer's  task  force  was 
provided  by  personnel  in  other  departmental  units  on 
a  participant  basis.  Thus,  the  control  system  task  force 
resident  engineer  was  able  to  draw  on  various 
expertise  available  within  the  Department  without 
concern  for  organizational  constraints. 

Several  benefits  of  the  task  force  concept  were 
achieved  over  those  of  the  Department's  previous  con- 
trol system  construction  methods.  First,  by  having 
electronic  control  system  specialists  instead  of  electri- 
cal inspectors  perform  the  inspection  work  during  the 
design,  manufacturing,  and  testing  phases  of  construc- 
tion, many  potential  design  and  maintenance 
problems  were  averted.  Secondly,  the  technician-in- 
spectors were  better  trained  for  their  ultimate  duty  of 
performing  maintenance  on  the  control  system  equip- 
ment. The  involvement  of  technicians  during  the  de- 
sign, manufacturing,  and  factory  testing  work  enabled 
them  to  obtain  first-hand  knowledge  of  the  system 
design.  Their  early  involvement  also  enabled  them  to 
obtain  actual  maintenance  experience  because  their 
checkout  testing  work  at  the  manufacturing  facility 
was  performed  while  the  system  was  suffering  from 
the  initial  failure  stage  of  its  component  parts.  Third- 
ly, the  control  system  engineering  staff  who  par- 
ticipated on  the  task  force,  but  were  not  directly 
responsible  for  construction  supervision,  obtained 
first-hand  knowledge  of  the  system  design  for  future 
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use  in  providing  engineering  support  to  the  operation 
and  maintenance  personnel.  Finally,  the  task  force 
staff  was  able  to  minimize  potential  misinterpreta- 
tions of  the  specification  during  the  design  and  manu- 
facturing phases  of  the'work  because  of  their  intimate 
knowledge  of  the  Department's  requirements  and  the 
intent  of  the  specifications. 

Plant  Control  System  Construction  Supervision 

Initially,  it  was  planned  to  supervise  construction  of 
the  pumping  plant  control  systems  as  special  fixtures. 
Much  of  the  factory  inspection  of  these  systems  was 
accomplished  in  this  manner.  By  early  1972,  however, 
only  one  of  the  systems  had  been  installed  in  the  field. 
With  the  regular  electrical-mechanical  work,  except 
for  the  control  systems,  near  completion  in  the  Project 
Offices  at  about  that  same  time,  the  Department 
elected  to  complete  the  remaining  factory  inspection, 
field  installation,  and  testing  of  the  pumping  plant 
control  systems  using  the  task  force  concept. 

Transition  to  the  task  force  concept  was  handled 
without  difficulty.  Control  system  personnel  who 
were  used  as  factory  inspectors  continued  on  the  task 
force  to  lend  continuity  to  the  plant  control  system 
testing  work. 

Also,  plant  control  system  design  engineers  for 
several  of  the  plant  control  systems  provided  technical 
supervision  of  some  of  the  field  testing  work.  This 
gave  continuity  to  the  engineering  evaluation  of  the 
systems  since  the  design  engineers  had  performed 
review  of  all  shop  drawings,  data,  and  materials 
during  the  design,  manufacturing,  factory  testing,  and 
installation  phases  of  the  work. 

Adopting  the  task  force  concept  for  plants  also 
provided  some  training  of  control  system  engineers 
and  technicians  for  future  engineering  support  of 
operation  and  maintenance. 

Control  System  Construction  Contracts 

The  project  control  system,  other  than  the  South 
Bay  Model,  was  constructed  under  12  separate  con- 
tracts. The  completion  of  additional  work  has  been 


performed  by  direct  procurements  of  equipment/ 
materiels  with  installation  under  service  agreements 
or  by  department  maintenance  personnel.  Table  17 
summarizes  the  major  plant  control  system  contracts, 
and  Table  18  summarizes  the  major  aqueduct  control 
system  contracts.  Only  those  contracts  in  excess  of 
$100,000  are  listed. 

Scope  of  the  Oroville-Thermalito  Control  System 
Contract 

Work  performed  under  Specification  No.  66-44 
consisted  of  furnishing  and  installing  the  supervisory 
systems  for  Edward  Hyatt  and  Thermalito 
Powerplants  and  for  Oroville  Switchyard;  the  alarm, 
events,  and  status  change  logging  equipment;  the 
automatic  load  control  system;  and  three  telemetry 
systems.  Maintenance  of  the  system  during  the 
operational  availability  demonstration  period,  as  well 
as  operation  and  maintenance  manuals,  were  also 
provided.  Training  in  programming,  maintenance, 
and  operation  of  the  system  was  included  in  the 
contract  work. 

Scope  of  the  Pumping  Plant  Control  System 
Contracts 

Work  performed  under  Specification  No.  65-41 
(Delta  Pumping  Plant)  included  furnishing  and 
installing  the  computer  system,  the  operator's  console, 
and  the  termination  cabinets  for  the  plant  to  the 
control  system  cabling.  The  uninterruptible  power 
supply  was  also  included.  Preparation  of  maintenance 
manuals  and  training  of  department  personnel  in  the 
programming,  maintenance,  and  operation  of  the 
system  were  also  included. 

Work  performed  under  Specifications  Nos.  68-09 
(Buena  Vista  and  Wheeler  Ridge  Pumping  Plants), 
68-35  (Wind  Gap  Pumping  Plant),  69-24  (A.  D. 
Edmonston  Pumping  Plant),  68-05  (Oso  Pumping 
Plant),  and  71-02  (Pearblossom  Pumping  Plant)  was 
essentially  the  same  as  at  the  Delta  Pumping  Plant 
except  that  the  work  included  furnishing  and 
installing     the     standard     communications     system 


Table 

17.     Summary  o 

Major  Plant  Co 

ntrol  System  Controcf  Data 

Delta 

Oroville- 
Thermalito 

Buena  Vista 

and 

Wheeler  Ridge 

Wind  Gap 

Oso 

A.  D. 

Kdmonstoii 

Pearblossom 

65-41 
55397,570 
55467,760 

5575,280 

12/22/65 

2/10/70 

General 

I'-lectric  Co. 

66-44 
551,247,500 
552,137,501 

55983,554 

10/17/66 

5/18/72 

Philco-Ford 

68-09 

55557,261 

55820,000 

(Est) 

55206,429- 

6/20/6K 

12/74  (Kst) 

Ruckcr  Co. 

68-35 
55458,898 
55533,692 

5578,747 
2/27/69 
8/2/73 
Litton  Systems 

68-50 
55535,000 
55601,438 

5585,113 

2/20/69 

10/3/73 

I'hilco-l'ord 

Litton  Systems 

69-24 

55889,000 

551,405,000 

(Kst) 

55403,239* 

4/27/70 

8/75  (Est) 

Astrodata 

Litton  Systems 

(Julton 

Industries 

71-02 

55561,970 

Final  Contract  Cost 

Total  Cost-Change  Orders.. 

55864,000 

(Kst) 

55272,023* 

5/20/71 

Completion  Date  .   . 

6/75  (Kst) 

Gulton 

Assignment  1 

Industries 

.\s  of  October  1974, 
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Table  18.     Summary  ol 

Major  Aqueduct  Control    System  Contract 

Data 

Alarm  and  Level 

Control-Delta  to 

7th  Std.  Rd. 

Delta  to  Bucna 

Vista  Pumping 

Plant  and  Coastal 

Branch 

Buena  Vista 

Pumping  Plant  to 

A.  D.  Edmonston 

Pumping  Plant 

POCC 
Wall  Displays 

So.  Calif. 
Area 

67-32 
}S224,761 
3281,431 

347,610 
6/8/67 
9/14/68 
Meva  Corp. 

68-08 
31,983,000 
32,519,090 

3419,486 
5/8/68 

10/14/71 
Fischbach  & 
Moore,  Inc. 

69-20 

3935,310 

31,156,607 

3189,173 
12/24/69 
4/30/72 
Fischbach  & 
Moore,  Inc. 

70-12 
3173,900 
3187,279 

86,604 
7/9/70 
1/20/72 
LFE  Corp. 

71-01 

32,087,000 

32,610,000 

(Est) 
3502,961» 

4/19/71 

12/74  (Est) 

LFE  Corp. 

•  As  of  October  1974. 


(excluding  the  modems)  required  for  interconnection 
to  the  ACC. 

The  SBIDAS  (Structural  Behavior  Instrumenta- 
tion Data  Acquisition  System)  was  also  included  as  a 
part  of  Specifications  Nos.  69-24  and  71-02. 

Scope  of  the  Aqueduct  Control  System  Contracts 

Work  performed  under  Specification  No.  67-32 
(California  Aqueduct  Alarm  Systems  and  Level 
Controls — Delta  to  7th  Standard  Road)  consisted  of 
furnishing  and  installing  analog  water-level 
controllers  and  water-level  sensing  equipment  at  26 
check  structures  of  the  California  Aqueduct  and 
Coastal  Branch  Aqueduct,  water-level  sensing 
equipment  at  three  pumping  plants  and  one  pump 
turnout,  and  alarm  annunciator  panels  at  the  Delta 
ACC  and  the  San  Luis  ACC. 

Work  performed  under  Specification  No.  68-08 
(California  Aqueduct  Control  System — Delta  to 
Buena  Vista  Pumping  Plant)  consisted  of  furnishing 
and  installing  the  control  system  for  that  portion  of 
the  California  Aqueduct  from  Clifton  Court  Forebay 
to  Buena  \'ista  Pumping  Plant  intake  and  the  Coastal 
Branch  Aqueduct. 

This  system  enables  the  portion  of  the  Aqueduct 
from  Clifton  Court  Forebay  to  O'Neill  Forebay  to  be 
monitored  and  controlled  from  the  Delta  ACC  located 
at  the  Delta  Operations  and  Maintenance  Center.  It 
provides  equipment  and  cabling  necessary  for  the 
Delta  ACC  and  provides  means  to  allow  the  control 
area  to  be  remotely  controlled  from  the  POCC. 
Included  in  these  controls  are  an  acoustic  velocity 
flowmeter  and  gate  controls  at  Clifton  Court  Forebay, 
12  check  structure  controls,  the  communications 
system  for  the  Delta  Pumping  Plant  control  system, 
and  electronic  interfacing  for  two  aqueduct  acoustic 
velocity  meters. 

The  San  Luis  control  area  originally  started  at 
O'Neill  Forebay  and  continued  to  Buena  \'ista 
Pumping  Plant  intake  and  included  the  Coastal 
Branch  Aqueduct.  The  control  system  in  this  reach 
provided  all  equipment  and  cabling  required  for 
remote  control  from  the  San  Luis  ACC,  located  at  the 


San  Luis  Pumping-Generating  Plant.  Provisions  to 
allow  remote  control  of  this  area  from  the  POCC  were 
also  included.  (Subsequent  to  the  completion  of  this 
contract,  the  Department  modified  the  San  Luis 
control  area  to  end  at  Kettleman  City.  The  facilities 
between  Kettleman  City  and  the  Buena  \'ista 
Pumping  Plant  intake  including  the  Coastal  Branch 
are  now  controlled  from  the  San  Joaquin  ACC.) 

Included  in  the  work  in  the  San  Luis  control  area 
was  the  furnishing  and  installing  of  control  and  in- 
strumentation equipment  at  18  check  structures  and 
two  noncomputerized  pumping  plants  (Las  Perillas 
and  Badger  Hill),  level  instrumentation  at  San  Luis 
Pumping-Generating  Plant  and  Dos  Amigos  Pump- 
ing Plant,  and  interfacing  for  an  aqueduct  acoustic 
velocity  meter. 

Work  also  included  the  furnishing  of  system 
maintenance  manuals,  maintenance  of  the  system 
while  it  was  being  tested,  and  training  of  department 
personnel. 

Work  performed  under  Specification  No.  69-20 
(California  Aqueduct  Control  System — Buena  \'ista 
Pumping  Plant  through  A.  D.  Edmonston  Pumping 
Plant)  consisted  of  furnishing  and  installing  the 
control  system  for  that  portion  of  the  California 
Aqueduct. 

This  system  enables  the  Aqueduct  to  be  monitored 
and  controlled  from  the  San  Joaquin  ACC  located  at 
the  San  Joaquin  Operations  and  Maintenance  Center 
near  Bakersfield.  Included  in  these  controls  are  seven 
check  structures  and  the  modems  for  four  pumping 
plant  control  systems. 

The  work  also  included  the  furnishing  of  systeni 
maintenance  manuals,  maintenance  of  the  system 
while  it  was  being  tested,  and  training  of  department 
personnel. 

Work  performed  under  Specification  No.  71-01 
(California  Aqueduct  Control  System — Southern 
California  Area)  consisted  of  furnishing  and 
installing  the  control  system  for  that  portion  of  the 
California  Aqueduct  from  A.  D.  Edmonston  Pumping 
Plant  surge  tank  to  Lake  Perris  on  the  East  Branch 
and  to  Castaic  Lake  on  the  West  Branch. 
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The  system  enables  the  portion  of  the  California 
Aqueduct  included  in  the  Southern  California  control 
area  to  be  controlled  from  the  Southern  California 
ACC  and  also  makes  provisions  to  allow  this  area  to 
be  controlled  from  the  POCC.  At  the  area  control 
center,  located  at  the  Castaic  Operations  and 
Maintenance  Center,  all  equipment  including  an 
uninterruptible  power  supply  and  cabling  were 
furnished  and  installed. 

Also  included  in  the  work  were  the  following  items: 
furnishing  and  installing  control  equipment  and 
cabling  for  10  two-gate  and  9  three-gate  check 
structures;  Quail  inlet  and  Quail  outlet  structures; 
Gorman  Creek  inlet  structure;  Cottonwood  chute 
gate;  Tehachapi  Crossing  complex;  Castaic,  Cedar 
Springs,  Pyramid,  and  Perris  Dams  and  Reservoirs; 
electronic  interfacing  of  Oso  and  Pearblossom 
Pumping  Plants;  Devil  Canyon  Afterbay  complex; 
and  Devil  Canyon  and  Castaic  Powerplant  controls 
(see  Appendix  B  for  general  location  of  these 
facilities). 

Included  in  the  work  was  furnishing  system 
maintenance  manuals,  maintenance  of  the  system 
while  it  was  being  tested,  and  training  of  department 
personnel. 

Work  performed  under  Specification  No.  70-12 
(Project  Operations  Control  Center  Wall  Displays 
and  Accessories)  consisted  of  furnishing  and 
installing  all  equipment  and  cabling  to  provide  a  wall 
display  of  the  Project  in  the  POCC  located  in  the 
Resources  Building,  Sacramento.  Included  in  the 
work  was  equipment  required  to  implement  the  area 


control  center  jurisdiction  system  throughout  the 
Project.  Maintenance  of  the  equipment  and  training 
of  department  personnel  in  maintenance  and 
operation  techniques  were  also  included. 

Complexity  of  Control  System 
Fabrication  and  Installation 

The  fabrication  and  installation  of  the  control 
system  components  under  the  various  contracts  was 
not  without  complications.  These  complications  were 
technical  and  administrative  in  nature.  Technical 
imperfections  encountered  on  initial  contracts 
prompted  revisions  in  the  specifications  for  later 
contracts.  Whenever  technical  problems  became 
apparent  during  contract  performance,  they  were 
resolved  either  by  contract  change  orders  or  by  the 
work  of  departmental  forces  following  completion  of 
the  contract. 

A  control  system  "Retrofit"  program  has  been  es- 
tablished for  department  personnel  to  perfect  and  in- 
tegrate the  individual  control  systems  for  optimum 
operational  effectiveness.  The  purposes  of  the  pro- 
gram are:  (1)  adjust  each  subsystem  on-site  (i.e.,  a 
plant  control  system  in  the  plant  control  room)  for 
operational  reliability  and  acceptability,  and  (2)  inter- 
connect each  subsystem  for  remote  operation  from  the 
ACC  or  the  POCC  with  comparable  operational  relia- 
bility and  acceptability.  Retrofit  work  requires  from 
one  to  two  years  for  completion  on  each  contract. 
Because  of  the  flexibility  inherent  in  a  computerized 
control  system,  the  required  adjustments  can  be  ac- 
complished mostly  by  software  modifications  with 
only  a  few  hardware  changes. 


CHAPTER  VII.     COMMUNICATION  SYSTEMS 


The  communication  systems  required  for  operation 
of  the  State  Water  Project  are  comprehensive  and 
extensive.  They  can  be  broadly  classified  as  data 
communications  and  voice  communications.  The 
various  communication  systems  are  used  at  all  times, 
and  the  Project's  operation  is  entirely  dependent  upon 
them.  They  are  one  of  the  most  critical  links  in  the 
project  control  system  and  require  a  high  degree  of 
reliability. 

The  entire  complex  of  communications  used  in  the 
control  system  are  leased  from  a  combine  of  telephone 
companies  within  California,  with  Pacific  Telephone 
and  Telegraph  Company  acting  as  spokesman  for  the 
industry.  The  mobile  VHP  radio  system  used  in  the 
general  operation  and  maintenance  of  the  Project  is  a 
state-owned  system. 

Planning  of  Communications 

In  1963,  communication  requirements  for  the  South 
Bay  Aqueduct  Control  System  Model  began  to 
finalize.  The  Department  prepared  preliminary 
designs  and  cost  studies  for  a  state-owned 
communications  system  to  satisfy  these  requirements. 
At  the  same  time,  the  Pacific  Telephone  and 
Telegraph  Company  prepared  a  proposal  for 
furnishing  the  necessary  circuits  and  facilities  under 
a  lease  arrangement.  A  comparison  of  lease  versus 
purchase  was  made,  and  the  results  indicated  it  was 
advantageous  to  utilize  leased  facilities  for  the  Model 
study.  Leasing  would  also  serve  as  a  test  of  this  method 
for  providing  the  communications  necessary  for  the 
entire  Project,  based  upon  the  Department's 
experience  on  the  South  Bay  Model.  Thus,  an 
agreement  for  lease  of  South  Bay  communications  was 
negotiated  between  the  State  and  Pacific  Telephone 
and  Telegraph  Company. 

In  early  1966,  the  Department  determined  that  an 
independent  opinion  on  the  question  of  leased  versus 
state-constructed  communication  facilities  would  be 
valuable  in  reaching  a  decision  on  furnishing  com- 
munications for  the  entire  Project.  Various  leading 
consulting  firms  in  the  field  of  communications  were 
selected  for  appraisal  by  the  Department.  After 
evaluation,  Messinger  Consultants  Company  was  se- 
lected to  conduct  the  study. 

The  Company  was  requested  to: 

1.  Investigate  the  total  communication  require- 
ments of  the  Project  which  would  include  but  not  be 
restricted  to  voice,  VHP  mobile  radio,  teletype  or 
other  written  messages,  and  telemetering  and  control 
links,  along  with  any  back-up  facilities  for  any  of  these 
items,  and  submit  a  comprehensive  plan  outlining  the 
total  communication  needs. 

2.  Investigate  the  alternative  methods  which  might 
be  used  to  meet  these  communication  needs  and  rec- 
ommend the  one  which  should  be  implemented. 

3.  Investigate  the  communication  personnel  re- 
quirements for  the  Project  and  make  a  recommenda- 


tion on  the  organization  and  staffing  required  to  carry 
out  the  recommended  plan  and  method  of  implemen- 
tation selected. 

4.  Review  and  provide  consulting  service  on  design 
of  the  communication  links  as  related  to  the  State 
Water  Project. 

5.  Assist  in  preparation  of  specifications  for 
procurement  of  communication  facilities  of  the 
Project  as  required. 

The  study  took  place  during  the  period  from  July  to 
December  1966,  and  a  report  was  presented  to  the 
Department  on  January  15,  1967  (Reference  9).  The 
Company  recommended  that  the  Department  lease 
the  necessary  communication  facilities  for  the  State 
Water  Project. 

Following  this  recommendation,  which  was  accept- 
ed by  the  Department,  negotiations  were  begun  for 
furnishing  the  necessary  communications.  Prior  to 
these  negotiations,  Pacific  Telephone  and  Telegraph 
Company  met  with  all  other  private  communication 
companies  of  the  State  whose  territory  the  Project 
would  pass  through  and  obtained  their  authority  to 
act  for  them  in  these  negotiations.  Participants  in 
these  negotiations  included  members  from  the  Pacific 
Telephone  and  Telegraph  Company,  Department  of 
General  Services,  and  Department  of  Water  Re- 
sources. Negotiations  resulted  in  an  agreement  which 
made  all  necessary  provisions  for  lease  of  data  and 
voice  communications  for  control  of  the  Project. 

Data  Communications  Design  Requirements 

Actual  design  and  construction  of  the  communica- 
tions system  was  done  by  the  telephone  industry.  The 
Department's  role  in  this  design  was  to  develop  the 
minimum  requirements  the  system  had  to  meet,  in- 
cluding provisions  for  future  expansion. 

When  remote  control  is  exercised  over  a  water 
delivery  system  as  large  and  complex  as  the  State 
Water  Project,  it  is  essential  that  it  be  available  to 
perform  its  task  at  all  times.  Any  downtime,  scheduled 
or  unscheduled,  during  which  control  points  cannot 
be  monitored  or  controlled,  carries  a  very  high  risk  of 
damage  to  the  facilities  and  their  surroundings,  as  well 
as  a  threat  to  the  safety  of  personnel. 

The  communications  system  is  the  most  vulnerable 
and  one  of  the  most  essential  links  in  the  reliability 
chain.  The  vulnerability  of  the  communication 
systems  arises  from  their  extended  nature,  which 
subjects  them  to  failure  and  physical  damage  at  places 
distant  from  department  facilities.  Their  remoteness 
makes  repair  difficult  and  time  consuming. 
Communication  systems  are  also  subject  to  the 
introduction  of  interfering  noise  from  the  atmosphere 
and  adjacent  electrical  or  communications  equipment. 

To  ensure  that  communications  would  be  reliable, 
the  Department  required  high  system  availability  of 
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service,  extensive  preoperational  testing,  early  detec- 
tion of  failures  to  reduce  the  mean-time-to-repair,  and 
special  maintenance  service.  In  addition,  a  right  to 
interconnect  department-owned  communications  was 
reserved  in  the  event  that  this  would  enhance  system 
reliability. 

System  Availability 

The  Department's  requirements  specified  an 
availability  of  service,  to  any  remote  site,  in  excess  of 
99.95%.  Dual  routing  of  data  communications  is  the 
principal  measure  to  assure  this  high  availability  of 
communications.  This  is  accomplished  by  providing 
two  geographically  separated  routes  from  an  ACC  or 
the  POCC  to  each  of  its  dependent  sites.  Figure  46 
shows  diagrammatically  the  arrangement  of  a  typical 
dual-routed  data  communication  line.  For  simplicity, 
the  diagram  shows  only  the  outgoing  direction  from 
the  control  centers  to  the  remote  sites;  however,  the 
lines  are  actually  four-wire,  full-duplex,  and  follow 
[j       the  same  route  in  both  directions. 

Preoperational  Testing 

Preoperational  testing  required  the  telephone 
industry  to  perform  extensive  tests  and  evaluation  of 
each  communication  line  prior  to  acceptance  by  the 
Department.  These  tests  were  designed  to  ensure  that 
any  problems  are  discovered  and  corrected  prior  to 
the  time  the  lines  are  put  into  service. 

Early  Detection  of  Failures 

Communication  failures  can  result  from  both 
department  equipment  failure  and  from 
ll  communication  line  failures.  It  is  important  to  be  able 
to  tell  the  difference  promptly.  Also,  because  of  the 
telephone  industries'  commitment  to  provide  a  high 
degree  of  reliability,  they  insisted  on  prompt  and 
automatic  alarms  when  their  lines  perform 
improperly. 

I  These  imperatives  resulted  in  an  investment  in 

II  equipment  and  computer  programs  to  monitor  the 
communication  lines  and  sense  communication 
failures  rapidly,  indicating  where  they  are,  both  by 
location  and  by  function  in  the  system. 

A  communication  line  monitor  is  a  system  installed 
and  operated  by  the  Department.  It  consists  of  line 

Ij  monitor  receivers  at  selected  remote  sites  which  close 
i  an  output  contact  when  digital  signals  are  present  at 
the  point  on  the  line  to  which  they  are  attached.  For 
any  line  segment,  one  line  monitor  is  placed  at  the 
extreme  sending  end  and  one  at  the  extreme  receiving 
end.  Thus,  if  the  sending  end  reports  messages  being 
sent  and  the  receiving  end  reports  no  messages  being 
received,  the  conclusion  is  that  the  line  has  failed  in 

»        between.  These  conditions  are  transmitted  directly  to 
j       the  responsible  telephone  company  as  relay  contact 
closures  to  terminals  provided  at  each  control  center. 
To  interpret  the  line  monitor  output  statuses,  it  is 
necessary  to  know  other  things  about  the  status  of  the 


system:  does  the  POCC  or  ACC  have  jurisdiction;  has 
power  failed  momentarily  or  permanently  at  a 
telephone  terminal;  or  is  there  evidence  of  failure  of 
telephone  cable  or  other  telephone  equipment  at  any 
location  along  the  way. 

To  supply  these  data,  the  control  system  conveys  to 
the  appropriate  control  center  and  then  to  the 
telephone  company's  alarm  system  a  power  failure 
alarm  and  a  "Telco"  alarm  from  each  remote  site.  The 
power  failure  alarm  answers  the  question  as  to 
whether  the  loss  of  data  was  caused  by  a  power  failure 
between  the  sensing  points.  The  "Telco"  alarm 
discloses  where  there  is  or  may  be  a  failure  of 
telephone  equipment. 

With  this  information,  the  responsible  telephone 
company  is  able  to  dispatch  a  repair  crew  sooner  and 
go  to  the  right  place,  assuring  prompt  repair  of 
failures  and  less  probability  of  simultaneous  failures 
on  both  communication  routes. 

Figure  47  is  a  block  diagram  of  a  typical  line 
monitor  system. 

Special  Maintenance  Service 

During  negotiations  with  Pacific  Telephone  and 
Telegraph  Company,  the  question  arose  regarding 
maintenance  service  of  the  circuits  which  would  be 
provided  to  the  Department.  Since  services  would 
span  the  entire  State  and  would  be  provided  by  many 
different  companies,  concern  was  expressed  regarding 
how  trouble  calls  would  be  directed  to  the  proper 
maintenance  organization.  A  Network  Control 
Office,  located  at  Pacific  Telephone  and  Telegraph 
Company  in  Sacramento,  was  developed  to  serve  as 
the  receiving  and  clearing  point  for  all  trouble  calls 
associated  with  project-leased  circuits.  This  Network 
Control  Office  assumes  the  responsibility  for 
directing  the  request  for  service  to  the  proper 
telephone  company  repair  group  within  the  State  on 
a  24-hour-per-day  basis. 

Interconnect  of  Parallel  Circuits 

By  agreement,  the  Department  reserved  the  right  to 
build  its  own  communication  facilities  to  augment 
those  of  the  carrier  companies  in  the  event  that  leased 
services  were  not  able  to  be  provided  reliably  or 
economically.  As  a  part  of  the  agreement,  the  State  can 
construct  its  own  circuits  if  another  parallel  circuit  is 
leased.  Because  the  leased  services  have  been  provided 
in  a  suitable  and  reliable  manner,  the  Department  has 
not  utilized  this  right. 

Voice  Communications 

To  aid  in  maintenance  of  the  extensive  and  complex 
project  control  systems,  it  was  required  that  a  voice 
communications  system  be  provided  which  would 
make  available  an  easy  and  convenient  method  of  oral 
communications  with  other  sites.  At  each  site  on  the 
Project  where  significant  amounts  of  electronic 
control    equipment    is    located,    a    voice    terminal. 


85 


86 


complete  with  speaker  and  head  set,  has  been 
provided.  The  circuit  is  arranged  in  a  party-line 
manner.  This  allows  two  or  more  of  the  sites  to  be 
connected  to  one  another  simultaneously.  This 
feature  is  used  for  the  coordination  required  between 
sites  during  testing  and  maintenance  of  the  control 
system. 

Since  these  systems  became  operational,  the  voice 
communication  circuits  have  been  widely  used  as 


operational  and  security  features  of  the  Project,  even 
more  than  originally  anticipated.  Other  sites,  where 
significant  amounts  of  electronic  equipment  do  not 
exist,  also  have  been  added  to  these  systems. 

\'oice  circuits  which  are  used  strictly  for 
operational  traffic  also  have  been  provided.  These  are 
dedicated  circuits  between  the  POCC  and  each  ACC 
and  between  the  POCC  and  various  outside  agencies. 
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APPENDIX  B 

ENGLISH  TO  METRIC  CONVERSIONS 
AND  PROJECT  STATISTICS 


CONVERSION  FACTORS 


English  to  Metric  System  of  Measurement 


Quantity 

English  unit 

Multiply  by 

To  get 
metric  equivalent 

Length 

inches 

2.54 

centimeters 

feet 

30.48 

centimeters 

0.3048 

meters 

0.0003048 

kilometers 

yards 

0.9144 

meters 

mi  les 

1,609.3 

meters 

1 .6093 

ki  lometers 

Area 

square  inches 

6.4516 

square  centimeters 

square  feet 

929.03 

square  centimeters 

square  yards 

0.83613 

square  meters 

acres 

0.40469 

hectares 

4,046.9 

square  meters 

0.0040469 

square  ki  lometers 

square  mi  les 

2.5898 

square  ki  lometers 

Volume 

gal  Ions 

3,785.4 

cubic  centimeters 

0.0037854 

cubic  meters 

3.7854 

liters 

a  ere -feet 

1,233.5 

cubic  meters 

1,233,500.0 

liters 

cubic  inches 

16.387 

cubic  centimeters 

cubic  feet 

0.028317 

cubic  meters 

cubic  yards 

0.76455 

cubic  meters 

764.55 

liters 

Velocity 

feet  per  second 

0.3048 

meters  per  second 

miles  per  hour 

1 .6093 

kilometers  per  hour 

Discharge 

cubic  feet  per  second 

or 
second-feet 

0.028317 

cubic  meters  per  second 

Weight 

pounds 

0.45359 

kilograms 

tons  (2,000  pounds) 

0.90718 

tons  (metric) 

Power 

horsepower 

0.7460 

kilowatts 
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